Jul 09 19:39:31.460977 ip-10-0-139-128 systemd[1]: Starting Kubernetes Kubelet... Jul 09 19:39:31.929083 ip-10-0-139-128 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 19:39:31.929083 ip-10-0-139-128 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 19:39:31.929083 ip-10-0-139-128 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 19:39:31.929602 ip-10-0-139-128 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 19:39:31.929602 ip-10-0-139-128 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 19:39:31.929602 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.929157 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 19:39:31.933867 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933852 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.933867 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933868 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933872 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933875 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933878 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933881 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933885 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933887 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933890 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933893 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933895 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933899 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933902 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933904 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933908 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933911 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933913 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933916 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933919 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933922 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933925 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.933929 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933928 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933930 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933933 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933936 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933939 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933942 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933945 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933948 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933950 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933953 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933956 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933958 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933961 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933964 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933967 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933969 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933972 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933975 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933978 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933980 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.934549 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933982 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933985 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933987 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933991 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933995 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.933999 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934002 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934005 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934007 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934010 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934012 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934015 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934017 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934020 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934023 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934026 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934032 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934036 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934039 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.935326 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934041 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934044 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934047 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934049 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934052 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934054 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934057 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934059 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934062 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934064 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934067 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934069 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934072 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934075 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934077 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934080 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934082 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934085 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934087 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934090 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.935869 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934092 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934095 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934097 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934099 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934102 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934104 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934107 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934110 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934113 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934117 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934120 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934123 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934125 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934128 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934131 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934134 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934136 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934139 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.936359 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.934142 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.937624 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937607 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.937624 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937624 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937628 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937632 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937635 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937637 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937640 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937643 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937646 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937648 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937651 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937669 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937672 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937675 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937677 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937680 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937682 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937685 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937690 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937694 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937697 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.937702 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937700 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937703 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937706 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937709 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937712 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937715 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937718 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937721 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937723 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937726 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937729 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937732 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937734 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937737 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937739 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937742 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937744 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937747 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937749 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.938228 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937752 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937755 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937757 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937760 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937762 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937765 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937768 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937770 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937773 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937775 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937777 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937780 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937783 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937785 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937788 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937791 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937794 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937797 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937801 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.938710 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937806 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937808 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937812 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937814 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937817 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937820 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937822 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937825 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937829 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937834 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937837 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937840 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937842 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937845 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937847 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937850 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937852 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937855 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937857 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937860 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.939165 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937863 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937866 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937868 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937871 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937875 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937877 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937880 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937883 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937886 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937888 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937890 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937893 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937895 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937898 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937900 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937903 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937906 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937908 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937911 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.937913 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.939635 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938339 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938344 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938347 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938350 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938352 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938355 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938357 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938360 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938363 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938366 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938369 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938371 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938374 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938377 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938380 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938382 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938385 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938388 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938390 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938393 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.940152 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938396 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938399 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938401 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938403 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938406 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938410 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938413 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938416 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938419 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938422 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938425 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938427 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938430 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938432 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938435 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938438 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938440 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938443 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938445 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938448 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.940631 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938450 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938453 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938455 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938458 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938461 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938464 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938466 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938469 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938471 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938474 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938477 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938479 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938482 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938484 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938486 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938489 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938491 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938494 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938496 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938499 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.941119 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938501 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938504 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938506 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938508 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938511 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938513 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938517 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938520 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938522 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938525 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938528 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938530 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938532 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938535 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938537 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938539 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938542 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938545 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938547 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938550 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.941594 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938552 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938555 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938559 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938562 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938566 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938570 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938573 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938576 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938578 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938581 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938584 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938587 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938589 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938592 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938594 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938597 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938599 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938602 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.938605 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.942083 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938711 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938725 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938732 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938737 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938741 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938744 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938748 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938752 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938755 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938758 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938762 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938765 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938768 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938771 2562 flags.go:64] FLAG: --cgroup-root="" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938774 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938777 2562 flags.go:64] FLAG: --client-ca-file="" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938781 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938783 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938788 2562 flags.go:64] FLAG: --cluster-domain="" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938791 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938794 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938797 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938800 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938809 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 19:39:31.942535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938812 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938815 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938819 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938822 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938824 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938827 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938830 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938833 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938837 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938840 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938843 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938846 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938848 2562 flags.go:64] FLAG: --enable-server="true" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938851 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938857 2562 flags.go:64] FLAG: --event-burst="100" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938860 2562 flags.go:64] FLAG: --event-qps="50" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938867 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938870 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938873 2562 flags.go:64] FLAG: --eviction-hard="" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938877 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938880 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938883 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938885 2562 flags.go:64] FLAG: --eviction-soft="" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938888 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938891 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 19:39:31.943121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938895 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938898 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938901 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938904 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938906 2562 flags.go:64] FLAG: --feature-gates="" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938910 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938913 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938916 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938919 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938922 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938925 2562 flags.go:64] FLAG: --help="false" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938928 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-128.ec2.internal" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938930 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938933 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938936 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938939 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938942 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938945 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938948 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938951 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938954 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938956 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938959 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938963 2562 flags.go:64] FLAG: --kube-reserved="" Jul 09 19:39:31.943796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938966 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938969 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938972 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938975 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938978 2562 flags.go:64] FLAG: --lock-file="" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938980 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938983 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938986 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938992 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938995 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.938998 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939001 2562 flags.go:64] FLAG: --logging-format="text" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939004 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939007 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939010 2562 flags.go:64] FLAG: --manifest-url="" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939013 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939017 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939020 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939024 2562 flags.go:64] FLAG: --max-pods="110" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939027 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939030 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939033 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939036 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939040 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939042 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 19:39:31.944387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939045 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939053 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939055 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939059 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939061 2562 flags.go:64] FLAG: --pod-cidr="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939064 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939069 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939073 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939076 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939079 2562 flags.go:64] FLAG: --port="10250" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939082 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939085 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05e7eced844f8f0d7" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939088 2562 flags.go:64] FLAG: --qos-reserved="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939091 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939094 2562 flags.go:64] FLAG: --register-node="true" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939097 2562 flags.go:64] FLAG: --register-with-taints="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939103 2562 flags.go:64] FLAG: --registry-burst="10" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939106 2562 flags.go:64] FLAG: --registry-qps="5" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939109 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939112 2562 flags.go:64] FLAG: --reserved-memory="" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939116 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939119 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939122 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939125 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939128 2562 flags.go:64] FLAG: --runonce="false" Jul 09 19:39:31.944996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939131 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939133 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939136 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939140 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939143 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939146 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939149 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939152 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939155 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939158 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939161 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939163 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939166 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939169 2562 flags.go:64] FLAG: --system-cgroups="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939173 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939179 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939182 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939185 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939189 2562 flags.go:64] FLAG: --tls-min-version="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939192 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939194 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939197 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939200 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939203 2562 flags.go:64] FLAG: --v="2" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939207 2562 flags.go:64] FLAG: --version="false" Jul 09 19:39:31.945596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939212 2562 flags.go:64] FLAG: --vmodule="" Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939216 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.939219 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939367 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939371 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939374 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939377 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939380 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939382 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939385 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939388 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939391 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939393 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939396 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939399 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939403 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939407 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939409 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939412 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939414 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.946257 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939417 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939422 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939424 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939428 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939432 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939435 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939438 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939441 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939444 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939447 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939449 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939452 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939455 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939458 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939461 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939464 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939466 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939468 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939471 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.946752 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939473 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939476 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939478 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939482 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939485 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939487 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939490 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939492 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939495 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939498 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939500 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939502 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939506 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939510 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939514 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939517 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939520 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939522 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939525 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.947246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939527 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939530 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939532 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939535 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939538 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939540 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939542 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939545 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939552 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939555 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939557 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939559 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939562 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939564 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939567 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939569 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939573 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939575 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939578 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939580 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.947733 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939583 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939585 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939588 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939590 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939592 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939595 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939597 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939601 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939604 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939606 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939609 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939611 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939614 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939616 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939619 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939621 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939624 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939626 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939629 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939631 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.948224 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939633 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939640 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939643 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.939646 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.940300 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.947455 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.947472 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.948289 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.948300 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948359 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948364 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948369 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948372 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948374 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948377 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.948744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948379 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948382 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948385 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948387 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948391 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948395 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948399 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948402 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948405 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948407 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948410 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948413 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948416 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948418 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948421 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948423 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948426 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948429 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948431 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.949115 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948434 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948437 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948439 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948442 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948446 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948449 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948452 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948455 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948457 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948460 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948462 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948465 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948467 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948470 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948472 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948475 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948477 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948480 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948482 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.949564 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948485 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948487 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948489 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948492 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948495 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948497 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948499 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948502 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948504 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948507 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948511 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948514 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948517 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948520 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948523 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948526 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948529 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948532 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948536 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948539 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.950029 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948542 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948544 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948547 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948550 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948552 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948554 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948557 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948559 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948562 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948564 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948567 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948569 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948572 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948574 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948576 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948579 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948581 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948584 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948586 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948589 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.950559 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948591 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948594 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948596 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948599 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948601 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948603 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948606 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948608 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948612 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948616 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948619 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948622 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948625 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948628 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948630 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.951109 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.948636 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948780 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948786 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948789 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948792 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948795 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948798 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948801 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948804 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948806 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948809 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948811 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948814 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948816 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948819 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948821 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948823 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948826 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948829 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948831 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 19:39:31.951474 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948834 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948836 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948838 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948841 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948845 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948849 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948852 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948854 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948857 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948860 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948862 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948865 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948868 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948870 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948873 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948876 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948878 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948880 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948883 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948885 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 19:39:31.952012 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948887 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948890 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948892 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948895 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948897 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948899 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948902 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948919 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948923 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948927 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948930 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948933 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948935 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948938 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948941 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948943 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948946 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948948 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948951 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 19:39:31.952500 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948954 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948956 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948959 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948961 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948964 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948966 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948969 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948971 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948974 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948976 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948979 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948981 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948983 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948986 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948988 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948991 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948993 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948995 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.948998 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949000 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 19:39:31.952975 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949002 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949005 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949007 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949013 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949015 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949018 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949020 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949023 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949025 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949028 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949032 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949035 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949038 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949042 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949044 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949047 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949050 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949053 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949056 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 19:39:31.953603 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949058 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:31.949061 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.949065 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.949174 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.951449 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.952934 2562 server.go:1019] "Starting client certificate rotation" Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.953052 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 19:39:31.954160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.953105 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 19:39:31.955720 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.955646 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 19:39:31.961199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.961177 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 19:39:31.973165 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.973145 2562 log.go:25] "Validated CRI v1 runtime API" Jul 09 19:39:31.975336 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.975321 2562 log.go:25] "Validated CRI v1 image API" Jul 09 19:39:31.976121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.976100 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 19:39:31.978862 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.978841 2562 fs.go:135] Filesystem UUIDs: map[09d15afb-d70d-4d71-b032-af48fdf6038f:/dev/nvme0n1p3 6e45aeeb-3fdc-4b1a-9258-44b121acfe31:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 09 19:39:31.978936 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.978860 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 19:39:31.984159 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.984018 2562 manager.go:216] Machine: {Timestamp:2026-07-09 19:39:31.982517147 +0000 UTC m=+0.386201067 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097783 MemoryCapacity:32812183552 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25e64d858b62efab6ea20ef67fb5e1 SystemUUID:ec25e64d-858b-62ef-ab6e-a20ef67fb5e1 BootID:80cc28b9-b9f0-4da9-b38f-191cf2a81e5d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406093824 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:53:12:f7:a4:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:53:12:f7:a4:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:d5:dd:79:02:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812183552 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 19:39:31.984159 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.984158 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 19:39:31.984270 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.984242 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 19:39:31.987761 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.987735 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 19:39:31.987897 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.987764 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 19:39:31.988348 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.988330 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 19:39:31.989390 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.989378 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 19:39:31.989429 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.989394 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 19:39:31.989429 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.989402 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 19:39:31.990177 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.990167 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 19:39:31.991978 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.991968 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 19:39:31.996526 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996514 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 19:39:31.996650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996640 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 19:39:31.996712 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996685 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 09 19:39:31.996712 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996705 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 19:39:31.996792 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996721 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 19:39:31.996792 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996732 2562 kubelet.go:398] "Adding apiserver pod source" Jul 09 19:39:31.996792 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.996742 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 19:39:31.997081 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.997070 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 19:39:31.997116 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.997089 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 19:39:31.999356 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:31.999340 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 19:39:32.001816 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.001801 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 19:39:32.001871 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.001834 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 19:39:32.002002 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.001990 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 19:39:32.002032 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002008 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 19:39:32.002032 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002014 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 19:39:32.002032 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002020 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 19:39:32.002032 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002029 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 19:39:32.002151 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002038 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 19:39:32.002596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002587 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 19:39:32.002628 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002596 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 19:39:32.002628 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002603 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 19:39:32.002628 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002609 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 19:39:32.002628 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002617 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 19:39:32.002628 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.002625 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 19:39:32.003713 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.003693 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 19:39:32.003849 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.003831 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 19:39:32.007742 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.007723 2562 server.go:1268] "Started kubelet" Jul 09 19:39:32.007843 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.007799 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 19:39:32.007887 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.007860 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 19:39:32.008588 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.008563 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 19:39:32.008944 ip-10-0-139-128 systemd[1]: Started Kubernetes Kubelet. Jul 09 19:39:32.011204 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.011182 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 19:39:32.011447 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.011426 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 19:39:32.011557 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.011534 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 19:39:32.011602 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.011566 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 19:39:32.012149 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.012126 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 09 19:39:32.013759 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.013739 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-psdmd" Jul 09 19:39:32.016519 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.016500 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 19:39:32.017336 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.017322 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 19:39:32.017423 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.017334 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 19:39:32.017423 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.017386 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.018140 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018124 2562 factory.go:55] Registering systemd factory Jul 09 19:39:32.018219 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018145 2562 factory.go:223] Registration of the systemd container factory successfully Jul 09 19:39:32.018557 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018544 2562 factory.go:153] Registering CRI-O factory Jul 09 19:39:32.018632 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018561 2562 factory.go:223] Registration of the crio container factory successfully Jul 09 19:39:32.018632 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018606 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 19:39:32.018632 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018623 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 19:39:32.018632 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018633 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 19:39:32.018848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018676 2562 factory.go:103] Registering Raw factory Jul 09 19:39:32.018848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018691 2562 manager.go:1196] Started watching for new ooms in manager Jul 09 19:39:32.018848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018726 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 19:39:32.018848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018766 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 09 19:39:32.018848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.018775 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 09 19:39:32.019099 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.019065 2562 manager.go:349] Starting recovery of all containers Jul 09 19:39:32.019515 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.018263 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-128.ec2.internal.18c0b65f19adb1f4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-128.ec2.internal,UID:ip-10-0-139-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-128.ec2.internal,},FirstTimestamp:2026-07-09 19:39:32.007678452 +0000 UTC m=+0.411362361,LastTimestamp:2026-07-09 19:39:32.007678452 +0000 UTC m=+0.411362361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-128.ec2.internal,}" Jul 09 19:39:32.023607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.023429 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-psdmd" Jul 09 19:39:32.025779 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.025751 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 19:39:32.026002 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.025968 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 19:39:32.031290 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.031272 2562 manager.go:354] Recovery completed Jul 09 19:39:32.034631 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.034610 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 09 19:39:32.037725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.037712 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.040061 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040045 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.040144 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040077 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.040144 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040092 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.040635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040620 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 19:39:32.040635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040634 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 19:39:32.040739 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040650 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 19:39:32.040739 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.040675 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.041540 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041525 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.041586 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041553 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.041586 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041563 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.041947 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041936 2562 policy_none.go:49] "None policy: Start" Jul 09 19:39:32.041987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041951 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 19:39:32.041987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.041961 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 19:39:32.042258 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.042192 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-128.ec2.internal.18c0b65f1b9bd2a5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-128.ec2.internal,UID:ip-10-0-139-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-128.ec2.internal,},FirstTimestamp:2026-07-09 19:39:32.040061605 +0000 UTC m=+0.443745513,LastTimestamp:2026-07-09 19:39:32.040061605 +0000 UTC m=+0.443745513,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-128.ec2.internal,}" Jul 09 19:39:32.043291 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.043280 2562 policy_none.go:47] "Start" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088354 2562 manager.go:337] "Starting Device Plugin manager" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.088405 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088421 2562 server.go:87] "Starting device plugin registration server" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088726 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088741 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088848 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088972 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.088981 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.090314 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 19:39:32.103289 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.090377 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.132199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.132153 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 19:39:32.133611 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.133587 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 19:39:32.133735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.133623 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 19:39:32.133735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.133649 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 19:39:32.133735 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.133703 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 19:39:32.135998 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.135979 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 19:39:32.189369 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.189279 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.190461 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.190441 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.190589 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.190484 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.190589 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.190499 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.190589 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.190529 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.197571 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.197553 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.197687 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.197578 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-128.ec2.internal\": node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.210355 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.210332 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.234800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.234763 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal"] Jul 09 19:39:32.234934 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.234836 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.235724 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.235706 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.235826 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.235739 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.235826 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.235750 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.237359 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.237346 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.237471 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.237457 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.237513 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.237484 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.238038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238013 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.238038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238035 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.238181 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238049 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.238181 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238057 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.238181 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238064 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.238181 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.238067 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.239383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.239367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.239447 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.239393 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 19:39:32.239988 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.239970 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientMemory" Jul 09 19:39:32.240067 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.240005 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 19:39:32.240067 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.240020 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeHasSufficientPID" Jul 09 19:39:32.255230 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.255215 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-128.ec2.internal\" not found" node="ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.258998 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.258984 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-128.ec2.internal\" not found" node="ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.311023 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.310990 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.411780 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.411743 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.420159 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.420124 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/934de4e9f3c9b88ff4abff78cc84b221-config\") pod \"kube-apiserver-proxy-ip-10-0-139-128.ec2.internal\" (UID: \"934de4e9f3c9b88ff4abff78cc84b221\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.420261 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.420169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.420261 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.420194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.512520 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.512450 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.520877 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.520939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/934de4e9f3c9b88ff4abff78cc84b221-config\") pod \"kube-apiserver-proxy-ip-10-0-139-128.ec2.internal\" (UID: \"934de4e9f3c9b88ff4abff78cc84b221\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.520939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.520939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.521056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/934de4e9f3c9b88ff4abff78cc84b221-config\") pod \"kube-apiserver-proxy-ip-10-0-139-128.ec2.internal\" (UID: \"934de4e9f3c9b88ff4abff78cc84b221\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.521056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.520970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7829272a3816f4829a67ddc721f3e3b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal\" (UID: \"e7829272a3816f4829a67ddc721f3e3b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.558039 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.558001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.562672 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.562594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:32.612914 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.612877 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.713486 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.713455 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.814141 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.814066 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.914705 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:32.914651 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:32.952213 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.952186 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 19:39:32.952824 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:32.952347 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 19:39:33.015609 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:33.015581 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:33.028333 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.028302 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 19:34:33 +0000 UTC" deadline="2028-01-03 23:19:50.852258095 +0000 UTC" Jul 09 19:39:33.028333 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.028327 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13035h40m17.823934063s" Jul 09 19:39:33.055455 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:33.055419 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934de4e9f3c9b88ff4abff78cc84b221.slice/crio-811b98d00acb8b1c2ef1389212f1616734249cebc17157fce49f1a061eef6fa8 WatchSource:0}: Error finding container 811b98d00acb8b1c2ef1389212f1616734249cebc17157fce49f1a061eef6fa8: Status 404 returned error can't find the container with id 811b98d00acb8b1c2ef1389212f1616734249cebc17157fce49f1a061eef6fa8 Jul 09 19:39:33.055686 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:33.055665 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7829272a3816f4829a67ddc721f3e3b.slice/crio-e1f9cc2d25b08050f15ee3565c29d23257462ab3b630dd118a3961607fe69ce1 WatchSource:0}: Error finding container e1f9cc2d25b08050f15ee3565c29d23257462ab3b630dd118a3961607fe69ce1: Status 404 returned error can't find the container with id e1f9cc2d25b08050f15ee3565c29d23257462ab3b630dd118a3961607fe69ce1 Jul 09 19:39:33.059421 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.059403 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:39:33.115934 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:33.115861 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:33.137806 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.137767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" event={"ID":"934de4e9f3c9b88ff4abff78cc84b221","Type":"ContainerStarted","Data":"811b98d00acb8b1c2ef1389212f1616734249cebc17157fce49f1a061eef6fa8"} Jul 09 19:39:33.138828 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.138804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" event={"ID":"e7829272a3816f4829a67ddc721f3e3b","Type":"ContainerStarted","Data":"e1f9cc2d25b08050f15ee3565c29d23257462ab3b630dd118a3961607fe69ce1"} Jul 09 19:39:33.156374 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.156348 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 19:39:33.216717 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:33.216677 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-128.ec2.internal\" not found" Jul 09 19:39:33.234892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.234866 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 19:39:33.317601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.317577 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" Jul 09 19:39:33.329297 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.329278 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 19:39:33.330415 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.330403 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" Jul 09 19:39:33.335922 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.335905 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 19:39:33.339679 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:33.339650 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 19:39:35.686744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.686716 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 19:39:35.810587 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.810536 2562 apiserver.go:52] "Watching apiserver" Jul 09 19:39:35.819412 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.819384 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 19:39:35.819850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.819823 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-glnk7","openshift-network-diagnostics/network-check-target-4xxq2","kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967","openshift-cluster-node-tuning-operator/tuned-npjlw","openshift-dns/node-resolver-4wtfh","openshift-image-registry/node-ca-j984v","openshift-network-operator/iptables-alerter-l9sv9","openshift-ovn-kubernetes/ovnkube-node-p8f5v","kube-system/global-pull-secret-syncer-9kgj4","kube-system/konnectivity-agent-56tk9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal","openshift-multus/multus-additional-cni-plugins-mrbsn","openshift-multus/multus-s8rsg"] Jul 09 19:39:35.821949 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.821927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.823549 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.823526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.824562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.824545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4gd8m\"" Jul 09 19:39:35.824697 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.824545 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.824697 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.824652 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 19:39:35.824697 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.824690 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.826770 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.826751 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 19:39:35.827033 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.827114 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827044 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 19:39:35.827114 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827101 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.827196 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pc7kv\"" Jul 09 19:39:35.827196 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827187 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.827333 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827264 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 19:39:35.827394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.827356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.828947 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.828928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.829861 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.829845 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 19:39:35.829985 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.829962 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.830054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.830035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sst4k\"" Jul 09 19:39:35.830144 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.830126 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.830231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.830216 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 19:39:35.830603 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.830300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5xzt2\"" Jul 09 19:39:35.830894 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.830877 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 19:39:35.831715 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.831698 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.831802 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.831717 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p8vtr\"" Jul 09 19:39:35.833104 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.832703 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.833104 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.833088 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.835397 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.835379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.835870 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.835852 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.835870 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.835863 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.836047 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.836026 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r5h4k\"" Jul 09 19:39:35.837208 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.837191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:35.837306 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:35.837283 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:35.838005 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.837988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 19:39:35.838086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.838045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rddvz\"" Jul 09 19:39:35.838152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.838109 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 19:39:35.838427 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.838292 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 19:39:35.838925 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.838906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:35.839022 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:35.838991 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:35.839211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.839192 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 19:39:35.839211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.839211 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.839211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.839217 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.840498 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.840482 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:35.840608 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:35.840539 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:35.842021 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.841996 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:35.842314 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-modprobe-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842314 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-lib-modules\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842314 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-node-log\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cni-binary-copy\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-socket-dir-parent\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e743b07-b293-4fb3-9e5e-982a16987054-host\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-var-lib-kubelet\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-kubelet\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-slash\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-var-lib-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-script-lib\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ft5\" (UniqueName: \"kubernetes.io/projected/9e743b07-b293-4fb3-9e5e-982a16987054-kube-api-access-66ft5\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842663 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dwk\" (UniqueName: \"kubernetes.io/projected/f5ee8c20-d161-49e4-ae02-87cb7e693f25-kube-api-access-c6dwk\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-tuned\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-bin\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6640aa51-5941-450a-9a3b-476c65042dc1-ovn-node-metrics-cert\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-netns\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-conf\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-multus\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzb7f\" (UniqueName: \"kubernetes.io/projected/6640aa51-5941-450a-9a3b-476c65042dc1-kube-api-access-bzb7f\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cnibin\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.842916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-kubelet\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-registration-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-kubernetes\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-run\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.842991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-hosts-file\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-systemd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-log-socket\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-env-overrides\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-sys-fs\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cnibin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-systemd\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e743b07-b293-4fb3-9e5e-982a16987054-serviceca\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpxq\" (UniqueName: \"kubernetes.io/projected/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-kube-api-access-hvpxq\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-os-release\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-bin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.843575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-conf-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-multus-certs\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-etc-kubernetes\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7t5\" (UniqueName: \"kubernetes.io/projected/6ce2c258-9f74-4932-a25d-380ee64b85e5-kube-api-access-xq7t5\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-tmp-dir\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-systemd-units\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-ovn\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-os-release\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-system-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-etc-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-hostroot\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843624 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-socket-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.844239 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysconfig\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-host\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-tmp\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-daemon-config\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsnh\" (UniqueName: \"kubernetes.io/projected/321848c0-bc4d-4b79-96c7-caa2c97d1580-kube-api-access-rbsnh\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843800 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt9d\" (UniqueName: \"kubernetes.io/projected/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-kube-api-access-sxt9d\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-sys\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-netns\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843901 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-config\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-system-cni-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.843985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-device-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.844043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.844072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-k8s-cni-cncf-io\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.844095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-netd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.844114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.845006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.844608 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:35.845727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.845019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-587lf\"" Jul 09 19:39:35.845727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.845187 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 19:39:35.845727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.845203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 19:39:35.847063 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.847046 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:39:35.847738 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.847719 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szg8g\"" Jul 09 19:39:35.847899 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.847879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 19:39:35.848002 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.847984 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 19:39:35.866315 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.866295 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5d2f2" Jul 09 19:39:35.873067 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.873041 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5d2f2" Jul 09 19:39:35.933275 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.933244 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 19:39:35.945404 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.945404 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-config\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.945404 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-system-cni-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-device-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-system-cni-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-k8s-cni-cncf-io\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-k8s-cni-cncf-io\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-netd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-device-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-netd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.945649 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-modprobe-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-lib-modules\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-node-log\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-node-log\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cni-binary-copy\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-socket-dir-parent\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-lib-modules\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-modprobe-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e743b07-b293-4fb3-9e5e-982a16987054-host\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-socket-dir-parent\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-iptables-alerter-script\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e743b07-b293-4fb3-9e5e-982a16987054-host\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-config\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946120 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqmg\" (UniqueName: \"kubernetes.io/projected/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-kube-api-access-fmqmg\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-d\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-var-lib-kubelet\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945962 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-kubelet\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-var-lib-kubelet\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.945992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-slash\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-kubelet\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-var-lib-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-script-lib\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ft5\" (UniqueName: \"kubernetes.io/projected/9e743b07-b293-4fb3-9e5e-982a16987054-kube-api-access-66ft5\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-host-slash\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dwk\" (UniqueName: \"kubernetes.io/projected/f5ee8c20-d161-49e4-ae02-87cb7e693f25-kube-api-access-c6dwk\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-tuned\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-var-lib-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-bin\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cni-binary-copy\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.946970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-cni-bin\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6640aa51-5941-450a-9a3b-476c65042dc1-ovn-node-metrics-cert\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-slash\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c95275d-ea82-412c-bdc1-101e3b1f7f73-konnectivity-ca\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-netns\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-netns\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-conf\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-multus\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946408 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzb7f\" (UniqueName: \"kubernetes.io/projected/6640aa51-5941-450a-9a3b-476c65042dc1-kube-api-access-bzb7f\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cnibin\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysctl-conf\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-dbus\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-multus\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.947778 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cnibin\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-ovnkube-script-lib\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-kubelet\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-kubelet\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-registration-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-kubernetes\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-registration-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-run\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-hosts-file\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-kubernetes\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-systemd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-run\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-hosts-file\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-systemd\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-log-socket\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-env-overrides\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-sys-fs\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-log-socket\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.948616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-sys-fs\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cnibin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-systemd\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e743b07-b293-4fb3-9e5e-982a16987054-serviceca\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946958 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-cnibin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.946980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-systemd\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpxq\" (UniqueName: \"kubernetes.io/projected/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-kube-api-access-hvpxq\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rp4n\" (UniqueName: \"kubernetes.io/projected/4035c42e-61d8-4047-86c7-8018eaecc828-kube-api-access-9rp4n\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-os-release\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-bin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-conf-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-multus-certs\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-etc-kubernetes\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7t5\" (UniqueName: \"kubernetes.io/projected/6ce2c258-9f74-4932-a25d-380ee64b85e5-kube-api-access-xq7t5\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6640aa51-5941-450a-9a3b-476c65042dc1-env-overrides\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-tmp-dir\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.949242 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-os-release\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-etc-kubernetes\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-systemd-units\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-var-lib-cni-bin\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947343 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-ovn\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-host-run-multus-certs\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-os-release\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-system-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e743b07-b293-4fb3-9e5e-982a16987054-serviceca\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947439 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-systemd-units\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-conf-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-etc-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-system-cni-dir\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.949997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-kubelet-config\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-os-release\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-etc-openvswitch\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-hostroot\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-socket-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysconfig\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-host\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ee8c20-d161-49e4-ae02-87cb7e693f25-hostroot\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-tmp\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/321848c0-bc4d-4b79-96c7-caa2c97d1580-socket-dir\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-host\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.947998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-daemon-config\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsnh\" (UniqueName: \"kubernetes.io/projected/321848c0-bc4d-4b79-96c7-caa2c97d1580-kube-api-access-rbsnh\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.952230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt9d\" (UniqueName: \"kubernetes.io/projected/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-kube-api-access-sxt9d\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c95275d-ea82-412c-bdc1-101e3b1f7f73-agent-certs\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-sys\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-netns\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-tmp-dir\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-host-run-netns\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-sys\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6640aa51-5941-450a-9a3b-476c65042dc1-run-ovn\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.948508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-sysconfig\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.949423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ee8c20-d161-49e4-ae02-87cb7e693f25-multus-daemon-config\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.950421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-etc-tuned\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.950472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ce2c258-9f74-4932-a25d-380ee64b85e5-tmp\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.952810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.951618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6640aa51-5941-450a-9a3b-476c65042dc1-ovn-node-metrics-cert\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.957702 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.957681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ft5\" (UniqueName: \"kubernetes.io/projected/9e743b07-b293-4fb3-9e5e-982a16987054-kube-api-access-66ft5\") pod \"node-ca-j984v\" (UID: \"9e743b07-b293-4fb3-9e5e-982a16987054\") " pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:35.958479 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.958451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzb7f\" (UniqueName: \"kubernetes.io/projected/6640aa51-5941-450a-9a3b-476c65042dc1-kube-api-access-bzb7f\") pod \"ovnkube-node-p8f5v\" (UID: \"6640aa51-5941-450a-9a3b-476c65042dc1\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:35.959243 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.959221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpxq\" (UniqueName: \"kubernetes.io/projected/bbd07248-df9c-4edd-bd0d-847cc00c7fa1-kube-api-access-hvpxq\") pod \"multus-additional-cni-plugins-mrbsn\" (UID: \"bbd07248-df9c-4edd-bd0d-847cc00c7fa1\") " pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:35.960081 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.960061 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dwk\" (UniqueName: \"kubernetes.io/projected/f5ee8c20-d161-49e4-ae02-87cb7e693f25-kube-api-access-c6dwk\") pod \"multus-s8rsg\" (UID: \"f5ee8c20-d161-49e4-ae02-87cb7e693f25\") " pod="openshift-multus/multus-s8rsg" Jul 09 19:39:35.962135 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.962111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7t5\" (UniqueName: \"kubernetes.io/projected/6ce2c258-9f74-4932-a25d-380ee64b85e5-kube-api-access-xq7t5\") pod \"tuned-npjlw\" (UID: \"6ce2c258-9f74-4932-a25d-380ee64b85e5\") " pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:35.962696 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.962677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsnh\" (UniqueName: \"kubernetes.io/projected/321848c0-bc4d-4b79-96c7-caa2c97d1580-kube-api-access-rbsnh\") pod \"aws-ebs-csi-driver-node-cn967\" (UID: \"321848c0-bc4d-4b79-96c7-caa2c97d1580\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:35.962850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:35.962835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt9d\" (UniqueName: \"kubernetes.io/projected/2dd2e12b-47b3-42ed-8296-3ecea75e2d26-kube-api-access-sxt9d\") pod \"node-resolver-4wtfh\" (UID: \"2dd2e12b-47b3-42ed-8296-3ecea75e2d26\") " pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:36.048423 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rp4n\" (UniqueName: \"kubernetes.io/projected/4035c42e-61d8-4047-86c7-8018eaecc828-kube-api-access-9rp4n\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:36.048538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-kubelet-config\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.048538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.048538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c95275d-ea82-412c-bdc1-101e3b1f7f73-agent-certs\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:36.048538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-iptables-alerter-script\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.048538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-kubelet-config\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqmg\" (UniqueName: \"kubernetes.io/projected/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-kube-api-access-fmqmg\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-host-slash\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c95275d-ea82-412c-bdc1-101e3b1f7f73-konnectivity-ca\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-host-slash\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.048659 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-dbus\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.048744 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.048769 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:36.548723669 +0000 UTC m=+3.139483876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:36.048812 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.048804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:36.54878655 +0000 UTC m=+3.139546743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:36.049182 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.048828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8afa780b-2966-4674-aa12-fe1b7b6e4a56-dbus\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.049182 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.049101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-iptables-alerter-script\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.049244 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.049189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c95275d-ea82-412c-bdc1-101e3b1f7f73-konnectivity-ca\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:36.051377 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.051356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c95275d-ea82-412c-bdc1-101e3b1f7f73-agent-certs\") pod \"konnectivity-agent-56tk9\" (UID: \"2c95275d-ea82-412c-bdc1-101e3b1f7f73\") " pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:36.057454 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.057431 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:36.057454 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.057449 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:36.057454 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.057459 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:36.057621 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.057508 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:39:36.557496337 +0000 UTC m=+3.148256520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:36.059332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.059309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rp4n\" (UniqueName: \"kubernetes.io/projected/4035c42e-61d8-4047-86c7-8018eaecc828-kube-api-access-9rp4n\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:36.060177 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.060160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqmg\" (UniqueName: \"kubernetes.io/projected/b44dfd67-2c2d-4fca-a897-55897f7fd5f4-kube-api-access-fmqmg\") pod \"iptables-alerter-l9sv9\" (UID: \"b44dfd67-2c2d-4fca-a897-55897f7fd5f4\") " pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.134475 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.134434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j984v" Jul 09 19:39:36.143054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.143030 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" Jul 09 19:39:36.152990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.152966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s8rsg" Jul 09 19:39:36.156041 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.155984 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd07248_df9c_4edd_bd0d_847cc00c7fa1.slice/crio-00e57a11e509e30f50995bf295b713052d14f074bba34e24156361c54c6dca9b WatchSource:0}: Error finding container 00e57a11e509e30f50995bf295b713052d14f074bba34e24156361c54c6dca9b: Status 404 returned error can't find the container with id 00e57a11e509e30f50995bf295b713052d14f074bba34e24156361c54c6dca9b Jul 09 19:39:36.157560 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.157543 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" Jul 09 19:39:36.163751 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.163724 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ee8c20_d161_49e4_ae02_87cb7e693f25.slice/crio-dca71a63b0f561b9619833181bb1122e3e8a0b7966c8bb3f1191658234476be8 WatchSource:0}: Error finding container dca71a63b0f561b9619833181bb1122e3e8a0b7966c8bb3f1191658234476be8: Status 404 returned error can't find the container with id dca71a63b0f561b9619833181bb1122e3e8a0b7966c8bb3f1191658234476be8 Jul 09 19:39:36.166483 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.166449 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-npjlw" Jul 09 19:39:36.172246 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.172222 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321848c0_bc4d_4b79_96c7_caa2c97d1580.slice/crio-76e750499d981d8702d769f901f6de6a053b94e61b537599eac7f51cd679e57c WatchSource:0}: Error finding container 76e750499d981d8702d769f901f6de6a053b94e61b537599eac7f51cd679e57c: Status 404 returned error can't find the container with id 76e750499d981d8702d769f901f6de6a053b94e61b537599eac7f51cd679e57c Jul 09 19:39:36.173189 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.173046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wtfh" Jul 09 19:39:36.180244 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.180226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:39:36.181049 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.181022 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce2c258_9f74_4932_a25d_380ee64b85e5.slice/crio-e3ec10a3bcbb7aa8ac37a200294f771deb227fa424ab4b032eb89b3c944a1245 WatchSource:0}: Error finding container e3ec10a3bcbb7aa8ac37a200294f771deb227fa424ab4b032eb89b3c944a1245: Status 404 returned error can't find the container with id e3ec10a3bcbb7aa8ac37a200294f771deb227fa424ab4b032eb89b3c944a1245 Jul 09 19:39:36.189174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.189157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:39:36.197132 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.197050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9sv9" Jul 09 19:39:36.207642 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.207615 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c95275d_ea82_412c_bdc1_101e3b1f7f73.slice/crio-c541fcf7796b15de4ba5b8b17d39f6f5e95afb0c547300d4af5157d6f8217f48 WatchSource:0}: Error finding container c541fcf7796b15de4ba5b8b17d39f6f5e95afb0c547300d4af5157d6f8217f48: Status 404 returned error can't find the container with id c541fcf7796b15de4ba5b8b17d39f6f5e95afb0c547300d4af5157d6f8217f48 Jul 09 19:39:36.208757 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:39:36.208595 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44dfd67_2c2d_4fca_a897_55897f7fd5f4.slice/crio-d9a88de5ed72beb490c8a27fefad37a5469d9fdd3fac9b4538842edf3c1a62d8 WatchSource:0}: Error finding container d9a88de5ed72beb490c8a27fefad37a5469d9fdd3fac9b4538842edf3c1a62d8: Status 404 returned error can't find the container with id d9a88de5ed72beb490c8a27fefad37a5469d9fdd3fac9b4538842edf3c1a62d8 Jul 09 19:39:36.554569 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.554491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:36.554738 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.554568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:36.554738 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.554705 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:36.554832 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.554759 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:37.554740825 +0000 UTC m=+4.145501023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:36.555121 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.555097 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:36.555215 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.555141 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:37.555130236 +0000 UTC m=+4.145890419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:36.655234 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.655106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:36.655401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.655286 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:36.655401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.655310 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:36.655401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.655323 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:36.655401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.655381 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:39:37.655362275 +0000 UTC m=+4.246122475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:36.875443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.875265 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 19:34:35 +0000 UTC" deadline="2028-01-31 16:11:36.594014779 +0000 UTC" Jul 09 19:39:36.875443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.875336 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13700h31m59.718683089s" Jul 09 19:39:36.948164 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.947723 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:36.948164 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:36.947837 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:36.960614 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.959847 2562 generic.go:356] "Generic (PLEG): container finished" podID="e7829272a3816f4829a67ddc721f3e3b" containerID="f20226bb6c39f10b0c640e54d00a8128e73c7fe70cad4e9efc0709851fb3f8bc" exitCode=0 Jul 09 19:39:36.960614 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.959927 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" event={"ID":"e7829272a3816f4829a67ddc721f3e3b","Type":"ContainerDied","Data":"f20226bb6c39f10b0c640e54d00a8128e73c7fe70cad4e9efc0709851fb3f8bc"} Jul 09 19:39:36.962802 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.962754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9sv9" event={"ID":"b44dfd67-2c2d-4fca-a897-55897f7fd5f4","Type":"ContainerStarted","Data":"d9a88de5ed72beb490c8a27fefad37a5469d9fdd3fac9b4538842edf3c1a62d8"} Jul 09 19:39:36.966880 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.966853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wtfh" event={"ID":"2dd2e12b-47b3-42ed-8296-3ecea75e2d26","Type":"ContainerStarted","Data":"34820eb2d52c611826ca5cd5d6fc1fb574b44633a5390ca590f9084d1a91e321"} Jul 09 19:39:36.971234 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.971206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-npjlw" event={"ID":"6ce2c258-9f74-4932-a25d-380ee64b85e5","Type":"ContainerStarted","Data":"e3ec10a3bcbb7aa8ac37a200294f771deb227fa424ab4b032eb89b3c944a1245"} Jul 09 19:39:36.973605 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.973562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" event={"ID":"321848c0-bc4d-4b79-96c7-caa2c97d1580","Type":"ContainerStarted","Data":"76e750499d981d8702d769f901f6de6a053b94e61b537599eac7f51cd679e57c"} Jul 09 19:39:36.976735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.976029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerStarted","Data":"00e57a11e509e30f50995bf295b713052d14f074bba34e24156361c54c6dca9b"} Jul 09 19:39:36.977688 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.977665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56tk9" event={"ID":"2c95275d-ea82-412c-bdc1-101e3b1f7f73","Type":"ContainerStarted","Data":"c541fcf7796b15de4ba5b8b17d39f6f5e95afb0c547300d4af5157d6f8217f48"} Jul 09 19:39:36.980162 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.980136 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"6d940ce3e2f7bc769983cb5a17f92a7d81556a35608c8c092be423e2ae2eb032"} Jul 09 19:39:36.984002 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.983958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8rsg" event={"ID":"f5ee8c20-d161-49e4-ae02-87cb7e693f25","Type":"ContainerStarted","Data":"dca71a63b0f561b9619833181bb1122e3e8a0b7966c8bb3f1191658234476be8"} Jul 09 19:39:36.986344 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:36.986296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j984v" event={"ID":"9e743b07-b293-4fb3-9e5e-982a16987054","Type":"ContainerStarted","Data":"334291ff2e743a01ab3d6a1c8e4d5d9bec42377d0a65f22888181a058720db22"} Jul 09 19:39:37.561700 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.561606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:37.561700 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.561682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:37.561926 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.561836 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:37.561998 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.561929 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:39.561882554 +0000 UTC m=+6.152642765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:37.562505 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.562484 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:37.562935 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.562915 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:39.562891309 +0000 UTC m=+6.153651498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:37.662469 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.662382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:37.662799 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.662776 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:37.662919 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.662804 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:37.662919 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.662818 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:37.662919 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.662881 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:39:39.662860687 +0000 UTC m=+6.253620892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:37.876008 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.875899 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 19:34:35 +0000 UTC" deadline="2028-02-13 13:09:23.239940569 +0000 UTC" Jul 09 19:39:37.876008 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.875961 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14009h29m45.363982999s" Jul 09 19:39:37.946938 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.946895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:37.947116 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.947028 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:37.947519 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:37.947494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:37.947643 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:37.947625 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:38.947054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:38.947017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:38.947573 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:38.947156 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:39.580249 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:39.580208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:39.580416 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:39.580303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:39.580455 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.580444 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:39.580520 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.580506 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:43.580487983 +0000 UTC m=+10.171248171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:39.580976 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.580922 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:39.580976 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.580975 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:43.580959457 +0000 UTC m=+10.171719645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:39.681709 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:39.681008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:39.681709 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.681221 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:39.681709 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.681244 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:39.681709 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.681256 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:39.681709 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.681313 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:39:43.681295159 +0000 UTC m=+10.272055355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:39.949326 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:39.948675 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:39.949326 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.948793 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:39.949326 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:39.949150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:39.949326 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:39.949238 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:40.947732 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:40.947691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:40.947931 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:40.947859 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:41.948111 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:41.947634 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:41.948111 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:41.947647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:41.948111 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:41.947746 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:41.948111 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:41.947833 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:42.958880 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:42.958843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:42.959316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:42.959029 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:43.616327 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:43.616289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:43.616534 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:43.616351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:43.616534 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.616473 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:43.616534 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.616474 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:43.616534 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.616535 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:51.61651707 +0000 UTC m=+18.207277268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:43.616771 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.616552 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:39:51.616543654 +0000 UTC m=+18.207303856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:43.717575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:43.717535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:43.717750 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.717711 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:43.717750 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.717733 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:43.717750 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.717743 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:43.717871 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.717802 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:39:51.717784852 +0000 UTC m=+18.308545035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:43.948057 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:43.947971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:43.948219 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.948087 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:43.948219 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:43.948143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:43.948331 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:43.948268 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:44.946907 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:44.946873 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:44.947359 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:44.947009 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:45.949079 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:45.949043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:45.949486 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:45.949166 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:45.949529 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:45.949518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:45.949635 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:45.949612 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:46.947731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:46.947693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:46.947899 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:46.947836 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:47.947060 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:47.947025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:47.947533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:47.947025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:47.947533 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:47.947163 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:47.947533 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:47.947213 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:48.946998 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:48.946962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:48.947150 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:48.947076 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:49.947694 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:49.947661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:49.948107 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:49.947789 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:49.948107 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:49.947858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:49.948107 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:49.947982 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:50.946909 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:50.946883 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:50.947114 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:50.947005 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:51.673964 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:51.673929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:51.674401 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:51.674001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:51.674401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.674103 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:51.674401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.674117 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:51.674401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.674168 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:07.674150551 +0000 UTC m=+34.264910737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:39:51.674401 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.674187 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:07.674178413 +0000 UTC m=+34.264938596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:39:51.775285 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:51.775251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:51.775448 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.775410 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:39:51.775448 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.775431 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:39:51.775448 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.775441 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:51.775615 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.775500 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:40:07.775481089 +0000 UTC m=+34.366241276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:39:51.950671 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:51.950571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:51.950820 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.950713 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:51.951121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:51.951091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:51.951246 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:51.951177 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:52.946997 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:52.946957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:52.947398 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:52.947095 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:53.947892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:53.947862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:53.948273 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:53.947958 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:53.948273 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:53.948050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:53.948273 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:53.948158 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:54.947655 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:54.947617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:54.947831 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:54.947734 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:55.947366 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:55.947044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:55.948129 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:55.947119 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:55.948129 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:55.947471 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:55.948129 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:55.947594 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:56.027433 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.027407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wtfh" event={"ID":"2dd2e12b-47b3-42ed-8296-3ecea75e2d26","Type":"ContainerStarted","Data":"42e122b41573070cb2c6e9b6ffc433b91ebe56f12f4fec8b31c15d49bd894287"} Jul 09 19:39:56.028809 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.028775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-npjlw" event={"ID":"6ce2c258-9f74-4932-a25d-380ee64b85e5","Type":"ContainerStarted","Data":"b07429be2b0b489af7465b3e17e071f2264ad43ec20e60cd0d9fc5ae6862c0e5"} Jul 09 19:39:56.030397 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.030363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" event={"ID":"321848c0-bc4d-4b79-96c7-caa2c97d1580","Type":"ContainerStarted","Data":"3b701299771ce4cb9d185edc49434b2b18f6f84946a5b145c23ff9ebe90b62ae"} Jul 09 19:39:56.031954 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.031929 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="41f6798bdc1bbd93eae3ba1b28d5c0685423194af6032cff9b22ec2915caa9dd" exitCode=0 Jul 09 19:39:56.032056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.032011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"41f6798bdc1bbd93eae3ba1b28d5c0685423194af6032cff9b22ec2915caa9dd"} Jul 09 19:39:56.033489 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.033465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56tk9" event={"ID":"2c95275d-ea82-412c-bdc1-101e3b1f7f73","Type":"ContainerStarted","Data":"4ab514dac582b82b8a7ddd4b48e46355011b7ba493f4762a8d991cd04a59b7f4"} Jul 09 19:39:56.035942 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.035916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"a191076a7616587fe9da3b6d5cc369b4eca493a95d8bb4bbd5a83b75ad10abd2"} Jul 09 19:39:56.036036 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.035970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"b63ded23464e41e59d476089acfbf9c7e983cc690a8c892a0bab5e1cbc1615d5"} Jul 09 19:39:56.036036 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.036011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"61a8f945dea35e3d270e89085f6558527bb8aaa14c9258b32f096343e6409dd1"} Jul 09 19:39:56.036138 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.036050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"617ccb42705e4d8edf53f73bd0108a97304ab5a581d3f7de362d1379de8691e3"} Jul 09 19:39:56.036138 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.036080 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"4ce3f46be2aa4d75c03f29dee6742dbee71cc21baf214189f05b91cd7e165c3b"} Jul 09 19:39:56.037260 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.037232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s8rsg" event={"ID":"f5ee8c20-d161-49e4-ae02-87cb7e693f25","Type":"ContainerStarted","Data":"714085e54f2d45152c50f29e5dcd75a0f1129024a868b7991618c0d38541c831"} Jul 09 19:39:56.038644 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.038624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j984v" event={"ID":"9e743b07-b293-4fb3-9e5e-982a16987054","Type":"ContainerStarted","Data":"f84d2e6f1f0f57f6f0582622ee647a08bffd55fcc3c758a675180b34c10c8c2b"} Jul 09 19:39:56.040019 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.039972 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" event={"ID":"934de4e9f3c9b88ff4abff78cc84b221","Type":"ContainerStarted","Data":"081af36991d827f8274def1997bd4cf199d59a366ab848d7a24d64f2eb3e36b1"} Jul 09 19:39:56.041666 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.041647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" event={"ID":"e7829272a3816f4829a67ddc721f3e3b","Type":"ContainerStarted","Data":"3788466d8a91a58ad1d31eec8dff1d1d6666d7acba7263c20616cef6a160c887"} Jul 09 19:39:56.044630 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.044569 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4wtfh" podStartSLOduration=2.88420244 podStartE2EDuration="22.044554162s" totalImagesPullingTime="19.160351722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.044278665 +0000 UTC m=+22.635038869" watchObservedRunningTime="2026-07-09 19:39:56.044554162 +0000 UTC m=+22.635314368" Jul 09 19:39:56.060886 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.060850 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-56tk9" podStartSLOduration=2.923081429 podStartE2EDuration="22.060841477s" totalImagesPullingTime="19.137760048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.060807566 +0000 UTC m=+22.651567789" watchObservedRunningTime="2026-07-09 19:39:56.060841477 +0000 UTC m=+22.651601683" Jul 09 19:39:56.076638 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.076603 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j984v" podStartSLOduration=11.078920133 podStartE2EDuration="22.076569225s" totalImagesPullingTime="10.997649092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.076438945 +0000 UTC m=+22.667199163" watchObservedRunningTime="2026-07-09 19:39:56.076569225 +0000 UTC m=+22.667329430" Jul 09 19:39:56.123944 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.123853 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-npjlw" podStartSLOduration=2.968214258 podStartE2EDuration="22.123839759s" totalImagesPullingTime="19.155625501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.123546578 +0000 UTC m=+22.714306815" watchObservedRunningTime="2026-07-09 19:39:56.123839759 +0000 UTC m=+22.714599964" Jul 09 19:39:56.170229 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.170178 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-128.ec2.internal" podStartSLOduration=21.170158539 podStartE2EDuration="21.170158539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.15031534 +0000 UTC m=+22.741075545" watchObservedRunningTime="2026-07-09 19:39:56.170158539 +0000 UTC m=+22.760918745" Jul 09 19:39:56.191967 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.191926 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s8rsg" podStartSLOduration=2.956392065 podStartE2EDuration="22.19191019s" totalImagesPullingTime="19.235518125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.191460517 +0000 UTC m=+22.782220748" watchObservedRunningTime="2026-07-09 19:39:56.19191019 +0000 UTC m=+22.782670396" Jul 09 19:39:56.192123 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.192096 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-128.ec2.internal" podStartSLOduration=21.192089397 podStartE2EDuration="21.192089397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:56.170300068 +0000 UTC m=+22.761060258" watchObservedRunningTime="2026-07-09 19:39:56.192089397 +0000 UTC m=+22.782849633" Jul 09 19:39:56.944153 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.944021 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 19:39:56.947633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:56.947613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:56.947918 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:56.947755 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:57.045259 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.045217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9sv9" event={"ID":"b44dfd67-2c2d-4fca-a897-55897f7fd5f4","Type":"ContainerStarted","Data":"f4a265e106143d38edd9617311f6091f5c5ad0a97ac4bd480c5e12edaaa18c03"} Jul 09 19:39:57.047358 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.047327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" event={"ID":"321848c0-bc4d-4b79-96c7-caa2c97d1580","Type":"ContainerStarted","Data":"7da19fcfcbb36d786d68cfe4cdd53f570cfeee47566f68f9555515885d0de030"} Jul 09 19:39:57.050933 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.050895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"cd4c3a78e59f1183343225eb5ae8a6189aad2b696c4cb8de3869de273a517340"} Jul 09 19:39:57.063751 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.063703 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l9sv9" podStartSLOduration=3.933482547 podStartE2EDuration="23.063684974s" totalImagesPullingTime="19.130202427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:57.063365176 +0000 UTC m=+23.654125394" watchObservedRunningTime="2026-07-09 19:39:57.063684974 +0000 UTC m=+23.654445183" Jul 09 19:39:57.912468 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.912366 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T19:39:56.944147886Z","UUID":"b315fd09-85c4-4968-8fe4-0ca14be63955","Handler":null,"Name":"","Endpoint":""} Jul 09 19:39:57.914139 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.914118 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 19:39:57.914271 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.914161 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 19:39:57.946927 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.946897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:57.947084 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:57.946902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:57.947084 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:57.947035 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:57.947196 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:57.947138 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:39:58.946916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:58.946888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:39:58.947383 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:58.946996 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:39:59.059245 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:59.059209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"67a283ce11db3a3aa3adb94ef7611555d433c091db0c51a7f0575dd2f5a19625"} Jul 09 19:39:59.061352 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:59.061314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" event={"ID":"321848c0-bc4d-4b79-96c7-caa2c97d1580","Type":"ContainerStarted","Data":"819a28ab427b249f8e8e1bd883cce21ed6cb37785e94276fc9df95e463febe56"} Jul 09 19:39:59.095784 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:59.095741 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn967" podStartSLOduration=3.627142428 podStartE2EDuration="25.095727441s" totalImagesPullingTime="21.468585013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:39:59.095480975 +0000 UTC m=+25.686241195" watchObservedRunningTime="2026-07-09 19:39:59.095727441 +0000 UTC m=+25.686487645" Jul 09 19:39:59.947299 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:59.947261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:39:59.947864 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:39:59.947315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:39:59.947864 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:59.947393 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:39:59.947864 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:39:59.947609 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:00.148383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:00.148342 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:40:00.149135 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:00.149110 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:40:00.946958 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:00.946759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:00.947151 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:00.947069 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:40:01.070047 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.070002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" event={"ID":"6640aa51-5941-450a-9a3b-476c65042dc1","Type":"ContainerStarted","Data":"e9fec1081f3fb69ae98aa8d758b43f440e2c9435518a196a6c66e64dfa9ab928"} Jul 09 19:40:01.070604 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.070328 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:40:01.071013 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.070993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-56tk9" Jul 09 19:40:01.099831 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.099793 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" podStartSLOduration=7.869507651 podStartE2EDuration="27.099778822s" totalImagesPullingTime="19.230271171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:01.099505266 +0000 UTC m=+27.690265489" watchObservedRunningTime="2026-07-09 19:40:01.099778822 +0000 UTC m=+27.690539008" Jul 09 19:40:01.947175 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.947138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:01.947340 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:01.947266 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:40:01.947340 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:01.947323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:01.947462 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:01.947430 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:02.073073 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.073032 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:02.073470 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.073134 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:02.073470 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.073177 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:02.090603 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.090558 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:02.095084 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.095059 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:02.380115 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.380085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kgj4"] Jul 09 19:40:02.380234 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.380224 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:02.380322 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:02.380305 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:02.383836 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.383810 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xxq2"] Jul 09 19:40:02.383943 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.383930 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:02.384066 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:02.384038 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:40:02.384340 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.384321 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-glnk7"] Jul 09 19:40:02.384424 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:02.384410 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:02.384520 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:02.384498 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:40:03.076087 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:03.075894 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="1e56de1c47fc048926dffe1957d2ed3c500cd392430af9327b0596c65e41b79b" exitCode=0 Jul 09 19:40:03.076573 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:03.075985 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"1e56de1c47fc048926dffe1957d2ed3c500cd392430af9327b0596c65e41b79b"} Jul 09 19:40:03.948352 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:03.948323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:03.948521 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:03.948369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:03.948521 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:03.948423 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:40:03.948521 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:03.948443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:03.948643 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:03.948522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:03.948643 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:03.948623 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:40:05.082637 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:05.082605 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="0eed72438eed74d9df1acf390e125722919739fe7d0783faf27f06a5fe20c94d" exitCode=0 Jul 09 19:40:05.083345 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:05.082665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"0eed72438eed74d9df1acf390e125722919739fe7d0783faf27f06a5fe20c94d"} Jul 09 19:40:05.947007 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:05.946981 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:05.947148 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:05.946981 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:05.947148 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:05.947109 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:40:05.947148 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:05.947124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:05.947301 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:05.947198 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:40:05.947301 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:05.947269 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:06.087756 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:06.087724 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="22639e596e1f841e3772b1182f3fecfd1744aa52b35fe0ee89ec5d9cc0712cf2" exitCode=0 Jul 09 19:40:06.088108 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:06.087767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"22639e596e1f841e3772b1182f3fecfd1744aa52b35fe0ee89ec5d9cc0712cf2"} Jul 09 19:40:07.692804 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.692606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:07.693241 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.692850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:07.693241 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.692760 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:40:07.693241 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.692947 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs podName:4035c42e-61d8-4047-86c7-8018eaecc828 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:39.692928807 +0000 UTC m=+66.283688990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs") pod "network-metrics-daemon-glnk7" (UID: "4035c42e-61d8-4047-86c7-8018eaecc828") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 19:40:07.693241 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.692966 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 19:40:07.693241 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.693025 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret podName:8afa780b-2966-4674-aa12-fe1b7b6e4a56 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:39.693005198 +0000 UTC m=+66.283765380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret") pod "global-pull-secret-syncer-9kgj4" (UID: "8afa780b-2966-4674-aa12-fe1b7b6e4a56") : object "kube-system"/"original-pull-secret" not registered Jul 09 19:40:07.793543 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.793504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:07.793722 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.793705 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 19:40:07.793797 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.793732 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 19:40:07.793797 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.793746 2562 projected.go:196] Error preparing data for projected volume kube-api-access-4fmbj for pod openshift-network-diagnostics/network-check-target-4xxq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:40:07.793892 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.793816 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj podName:91030579-d0bd-4adf-9004-dc7810e04abb nodeName:}" failed. No retries permitted until 2026-07-09 19:40:39.793796653 +0000 UTC m=+66.384556838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fmbj" (UniqueName: "kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj") pod "network-check-target-4xxq2" (UID: "91030579-d0bd-4adf-9004-dc7810e04abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 19:40:07.947537 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.947422 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:07.947537 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.947407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:07.947753 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:07.947423 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:07.947753 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.947557 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kgj4" podUID="8afa780b-2966-4674-aa12-fe1b7b6e4a56" Jul 09 19:40:07.947863 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.947793 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glnk7" podUID="4035c42e-61d8-4047-86c7-8018eaecc828" Jul 09 19:40:07.947863 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:07.947795 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xxq2" podUID="91030579-d0bd-4adf-9004-dc7810e04abb" Jul 09 19:40:09.246114 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.246027 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-128.ec2.internal" event="NodeReady" Jul 09 19:40:09.246602 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.246174 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 19:40:09.282772 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.282335 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:40:09.285133 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.285107 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.286460 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.285765 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8"] Jul 09 19:40:09.287563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.287506 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb"] Jul 09 19:40:09.287859 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.287818 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.288211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.288184 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 19:40:09.288339 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.288315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 19:40:09.288427 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.288410 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpggp\"" Jul 09 19:40:09.289325 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.289295 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-svwdq"] Jul 09 19:40:09.289422 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.289386 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" Jul 09 19:40:09.290093 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.290072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 19:40:09.291124 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.291104 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s"] Jul 09 19:40:09.291227 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.291210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.293025 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.293001 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt"] Jul 09 19:40:09.293125 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.293111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.295770 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.295158 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn"] Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.295967 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.296138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.296572 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.297355 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.297603 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xxgrh\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.298229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5tgr8\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.298709 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.299067 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.300144 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.300650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.300331 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.303180 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.303156 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 19:40:09.303690 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.303672 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 19:40:09.304305 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304284 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.304385 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304334 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nz9x5\"" Jul 09 19:40:09.304385 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304347 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 19:40:09.304490 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 19:40:09.304490 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304458 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.304613 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304528 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.304934 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.304913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" Jul 09 19:40:09.305552 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.305530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:40:09.305667 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.305619 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 19:40:09.305729 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.305666 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-spvwb\"" Jul 09 19:40:09.305837 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.305823 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.307103 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.307084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb"] Jul 09 19:40:09.307976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.307905 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8"] Jul 09 19:40:09.311644 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.311627 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 19:40:09.311863 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.311738 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.314676 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.314656 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 19:40:09.316399 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.316382 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w77vk\"" Jul 09 19:40:09.316513 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.316499 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.316653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.316634 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hxrkd\"" Jul 09 19:40:09.316723 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.316689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn"] Jul 09 19:40:09.317376 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.316981 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.317533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.317517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-svwdq"] Jul 09 19:40:09.318175 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.318151 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt"] Jul 09 19:40:09.318741 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.318727 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l6qr9"] Jul 09 19:40:09.321782 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.321760 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s"] Jul 09 19:40:09.321936 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.321916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.322448 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.322426 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w56d7"] Jul 09 19:40:09.324825 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.324808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.325000 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.324980 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 19:40:09.325134 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.325096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6qw7b\"" Jul 09 19:40:09.325483 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.325464 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 19:40:09.325814 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.325788 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 19:40:09.327288 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.327269 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 19:40:09.328245 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.328229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 19:40:09.328755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.328734 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtfn7\"" Jul 09 19:40:09.335327 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.335302 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6qr9"] Jul 09 19:40:09.342775 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.342754 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w56d7"] Jul 09 19:40:09.404626 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbv7\" (UniqueName: \"kubernetes.io/projected/38e13fea-c6f3-4bec-bccd-9a976004b5bc-kube-api-access-qdbv7\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38e13fea-c6f3-4bec-bccd-9a976004b5bc-serving-cert\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-trusted-ca\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98146308-3725-4523-80ec-7d626c3aee7d-config-volume\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-config\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.404827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgcq\" (UniqueName: \"kubernetes.io/projected/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-kube-api-access-2zgcq\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ee5626-3a98-4d0d-9667-fd294bea5c26-serving-cert\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9b9\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknk5\" (UniqueName: \"kubernetes.io/projected/c9ee5626-3a98-4d0d-9667-fd294bea5c26-kube-api-access-nknk5\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.404992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrt8p\" (UniqueName: \"kubernetes.io/projected/d488f561-09b6-48a7-af35-5805348a74f9-kube-api-access-wrt8p\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbgk\" (UniqueName: \"kubernetes.io/projected/98146308-3725-4523-80ec-7d626c3aee7d-kube-api-access-slbgk\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glr2g\" (UniqueName: \"kubernetes.io/projected/6b7a3330-d7dc-4363-b63b-93d50adc18dc-kube-api-access-glr2g\") pod \"volume-data-source-validator-78495597f6-njtmb\" (UID: \"6b7a3330-d7dc-4363-b63b-93d50adc18dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2k9\" (UniqueName: \"kubernetes.io/projected/e4869655-d404-4b5c-99b4-9131c36232a9-kube-api-access-dp2k9\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d488f561-09b6-48a7-af35-5805348a74f9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-cert\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.405438 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.405755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405448 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98146308-3725-4523-80ec-7d626c3aee7d-tmp-dir\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.405755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ee5626-3a98-4d0d-9667-fd294bea5c26-config\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.405755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgmn\" (UniqueName: \"kubernetes.io/projected/e0b5ada2-f474-4154-98c0-fbd8db86205e-kube-api-access-zvgmn\") pod \"network-check-source-869dcfcdcd-zfmhn\" (UID: \"e0b5ada2-f474-4154-98c0-fbd8db86205e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" Jul 09 19:40:09.405755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.405529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d488f561-09b6-48a7-af35-5805348a74f9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.506482 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9b9\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.506482 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknk5\" (UniqueName: \"kubernetes.io/projected/c9ee5626-3a98-4d0d-9667-fd294bea5c26-kube-api-access-nknk5\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.506775 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrt8p\" (UniqueName: \"kubernetes.io/projected/d488f561-09b6-48a7-af35-5805348a74f9-kube-api-access-wrt8p\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.506775 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.506775 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.506639 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:09.506775 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.506653 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:09.506775 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.506718 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:10.006699596 +0000 UTC m=+36.597459795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:09.507001 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbgk\" (UniqueName: \"kubernetes.io/projected/98146308-3725-4523-80ec-7d626c3aee7d-kube-api-access-slbgk\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.507001 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glr2g\" (UniqueName: \"kubernetes.io/projected/6b7a3330-d7dc-4363-b63b-93d50adc18dc-kube-api-access-glr2g\") pod \"volume-data-source-validator-78495597f6-njtmb\" (UID: \"6b7a3330-d7dc-4363-b63b-93d50adc18dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" Jul 09 19:40:09.507001 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507001 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.506957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2k9\" (UniqueName: \"kubernetes.io/projected/e4869655-d404-4b5c-99b4-9131c36232a9-kube-api-access-dp2k9\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.507152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d488f561-09b6-48a7-af35-5805348a74f9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.507152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507232 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507232 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507232 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-cert\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.507373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.507373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.507373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98146308-3725-4523-80ec-7d626c3aee7d-tmp-dir\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ee5626-3a98-4d0d-9667-fd294bea5c26-config\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgmn\" (UniqueName: \"kubernetes.io/projected/e0b5ada2-f474-4154-98c0-fbd8db86205e-kube-api-access-zvgmn\") pod \"network-check-source-869dcfcdcd-zfmhn\" (UID: \"e0b5ada2-f474-4154-98c0-fbd8db86205e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d488f561-09b6-48a7-af35-5805348a74f9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbv7\" (UniqueName: \"kubernetes.io/projected/38e13fea-c6f3-4bec-bccd-9a976004b5bc-kube-api-access-qdbv7\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38e13fea-c6f3-4bec-bccd-9a976004b5bc-serving-cert\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-trusted-ca\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.507562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98146308-3725-4523-80ec-7d626c3aee7d-config-volume\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-config\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgcq\" (UniqueName: \"kubernetes.io/projected/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-kube-api-access-2zgcq\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ee5626-3a98-4d0d-9667-fd294bea5c26-serving-cert\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.507910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.507751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d488f561-09b6-48a7-af35-5805348a74f9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.508219 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.508085 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.508219 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.508188 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:09.508301 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.508245 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:10.008229052 +0000 UTC m=+36.598989269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:09.508350 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.508301 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:09.508350 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.508322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98146308-3725-4523-80ec-7d626c3aee7d-config-volume\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.508533 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:09.508329 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:10.008320496 +0000 UTC m=+36.599080678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:09.508699 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.508676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-config\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.509023 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.508994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c9ee5626-3a98-4d0d-9667-fd294bea5c26-config\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.509090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.509037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e13fea-c6f3-4bec-bccd-9a976004b5bc-trusted-ca\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.509269 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.509228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98146308-3725-4523-80ec-7d626c3aee7d-tmp-dir\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.512384 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.512339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9ee5626-3a98-4d0d-9667-fd294bea5c26-serving-cert\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.512712 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.512669 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.512712 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.512697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d488f561-09b6-48a7-af35-5805348a74f9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.512981 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.512923 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38e13fea-c6f3-4bec-bccd-9a976004b5bc-serving-cert\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.513813 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.513762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.517258 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.517236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknk5\" (UniqueName: \"kubernetes.io/projected/c9ee5626-3a98-4d0d-9667-fd294bea5c26-kube-api-access-nknk5\") pod \"service-ca-operator-7695fc6c8c-9zlrt\" (UID: \"c9ee5626-3a98-4d0d-9667-fd294bea5c26\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.518006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.517978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2k9\" (UniqueName: \"kubernetes.io/projected/e4869655-d404-4b5c-99b4-9131c36232a9-kube-api-access-dp2k9\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:09.518082 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.518014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glr2g\" (UniqueName: \"kubernetes.io/projected/6b7a3330-d7dc-4363-b63b-93d50adc18dc-kube-api-access-glr2g\") pod \"volume-data-source-validator-78495597f6-njtmb\" (UID: \"6b7a3330-d7dc-4363-b63b-93d50adc18dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" Jul 09 19:40:09.518426 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.518361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbv7\" (UniqueName: \"kubernetes.io/projected/38e13fea-c6f3-4bec-bccd-9a976004b5bc-kube-api-access-qdbv7\") pod \"console-operator-7866497494-svwdq\" (UID: \"38e13fea-c6f3-4bec-bccd-9a976004b5bc\") " pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.518882 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.518859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9b9\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.519143 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.518889 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgcq\" (UniqueName: \"kubernetes.io/projected/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-kube-api-access-2zgcq\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.520042 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.519955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbgk\" (UniqueName: \"kubernetes.io/projected/98146308-3725-4523-80ec-7d626c3aee7d-kube-api-access-slbgk\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:09.520275 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.520214 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2-cert\") pod \"ingress-canary-l6qr9\" (UID: \"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2\") " pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.520417 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.520396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrt8p\" (UniqueName: \"kubernetes.io/projected/d488f561-09b6-48a7-af35-5805348a74f9-kube-api-access-wrt8p\") pod \"kube-storage-version-migrator-operator-799cb7f958-ch6g8\" (UID: \"d488f561-09b6-48a7-af35-5805348a74f9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.521110 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.521091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgmn\" (UniqueName: \"kubernetes.io/projected/e0b5ada2-f474-4154-98c0-fbd8db86205e-kube-api-access-zvgmn\") pod \"network-check-source-869dcfcdcd-zfmhn\" (UID: \"e0b5ada2-f474-4154-98c0-fbd8db86205e\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" Jul 09 19:40:09.522102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.522021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:09.612991 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.612958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" Jul 09 19:40:09.624444 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.624417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" Jul 09 19:40:09.631490 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.631467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" Jul 09 19:40:09.640153 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.640135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:09.656930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.656907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" Jul 09 19:40:09.663157 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.663129 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6qr9" Jul 09 19:40:09.947191 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.947102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:09.947487 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.947104 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:09.947777 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.947116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:09.950354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.950336 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 19:40:09.950451 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.950388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 19:40:09.951745 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.951726 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-95dhd\"" Jul 09 19:40:09.951961 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:09.951945 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mczbz\"" Jul 09 19:40:10.011932 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.011891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:10.012105 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.011948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:10.012105 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012070 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:10.012105 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012088 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:10.012254 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012137 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:11.012121668 +0000 UTC m=+37.602881855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:10.012254 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012206 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:11.012183837 +0000 UTC m=+37.602944035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:10.012341 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.012253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:10.012422 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012400 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:10.012422 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012420 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:10.012661 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:10.012471 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:11.012455055 +0000 UTC m=+37.603215242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:10.338622 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.338572 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt"] Jul 09 19:40:10.343351 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:10.343318 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ee5626_3a98_4d0d_9667_fd294bea5c26.slice/crio-f367e8bd72f931e8a21e32a6af6957607a388a3538a1efd5317490e7c751c616 WatchSource:0}: Error finding container f367e8bd72f931e8a21e32a6af6957607a388a3538a1efd5317490e7c751c616: Status 404 returned error can't find the container with id f367e8bd72f931e8a21e32a6af6957607a388a3538a1efd5317490e7c751c616 Jul 09 19:40:10.345722 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.345693 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6qr9"] Jul 09 19:40:10.353571 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.351873 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8"] Jul 09 19:40:10.353571 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.351935 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-svwdq"] Jul 09 19:40:10.353874 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.353792 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb"] Jul 09 19:40:10.354455 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.354436 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn"] Jul 09 19:40:10.737775 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.737679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:10.846719 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:10.846679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:11.023452 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.023364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:11.023639 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.023449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:11.023639 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.023494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:11.023790 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.023771 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:11.023850 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.023794 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:11.023902 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.023858 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:13.023837886 +0000 UTC m=+39.614598093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:11.024362 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.024278 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:11.024362 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.024331 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:13.02431569 +0000 UTC m=+39.615075879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:11.024713 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.024638 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:11.024713 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:11.024691 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:13.024676759 +0000 UTC m=+39.615436956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:11.107752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.107523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" event={"ID":"e0b5ada2-f474-4154-98c0-fbd8db86205e","Type":"ContainerStarted","Data":"25116602ce85f7efe0c08f5f5c60440db04f1c7ce6b1d3bcee8da85ba994f778"} Jul 09 19:40:11.111670 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.111611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-svwdq" event={"ID":"38e13fea-c6f3-4bec-bccd-9a976004b5bc","Type":"ContainerStarted","Data":"38ee53665aa2859f11baacbb7184a9327dfe60f0e2fffc86dc93009549623223"} Jul 09 19:40:11.115165 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.115086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" event={"ID":"d488f561-09b6-48a7-af35-5805348a74f9","Type":"ContainerStarted","Data":"33b30fcd56f55f3b509922c9996c02cd22b4a648073e980931bdb9b93c65ba6f"} Jul 09 19:40:11.118442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.118366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6qr9" event={"ID":"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2","Type":"ContainerStarted","Data":"ad57b03695ba73d8d01e7b7246c0fae4bcd8c53ee6d46c443d8b19cc03f8bd20"} Jul 09 19:40:11.129900 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.128685 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="439b11b6de7efdfac92ba1eb281325bad6146038de0d9a58113423d07e106c71" exitCode=0 Jul 09 19:40:11.129900 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.129673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"439b11b6de7efdfac92ba1eb281325bad6146038de0d9a58113423d07e106c71"} Jul 09 19:40:11.133727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.133654 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" event={"ID":"6b7a3330-d7dc-4363-b63b-93d50adc18dc","Type":"ContainerStarted","Data":"2d3b1a50f3d23822013f8ff20b72f73ee704b4f4a92e3f94338c24733e92ff1c"} Jul 09 19:40:11.138399 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:11.138320 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" event={"ID":"c9ee5626-3a98-4d0d-9667-fd294bea5c26","Type":"ContainerStarted","Data":"f367e8bd72f931e8a21e32a6af6957607a388a3538a1efd5317490e7c751c616"} Jul 09 19:40:12.149607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:12.149552 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd07248-df9c-4edd-bd0d-847cc00c7fa1" containerID="c14fc1a4e3577fbd2e9e626dce500a580c5082dd51ba14be167aaa6d3812f140" exitCode=0 Jul 09 19:40:12.150323 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:12.149650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerDied","Data":"c14fc1a4e3577fbd2e9e626dce500a580c5082dd51ba14be167aaa6d3812f140"} Jul 09 19:40:12.366234 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:12.366176 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:12.457087 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:12.456993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:12.557534 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:12.557420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:13.045026 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.044924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:13.045178 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.045024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:13.045178 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.045062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:13.045178 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045118 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:13.045178 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045140 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:13.045178 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045174 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:13.045388 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045204 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:17.045183622 +0000 UTC m=+43.635943809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:13.045388 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045227 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:17.045211143 +0000 UTC m=+43.635971346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:13.045388 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045268 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:13.045388 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:13.045296 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:17.045286315 +0000 UTC m=+43.636046499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:13.156696 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.156655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" event={"ID":"bbd07248-df9c-4edd-bd0d-847cc00c7fa1","Type":"ContainerStarted","Data":"341c7057ac1beff821d5cbc0361fab799ab2339b1415b18a2281d0d8470839f4"} Jul 09 19:40:13.159286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.159232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" event={"ID":"6b7a3330-d7dc-4363-b63b-93d50adc18dc","Type":"ContainerStarted","Data":"6ba6989969fc5ccb6d5024d58c05d309399ccf7fa0cfe0933cc19f75a35ea220"} Jul 09 19:40:13.181721 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.181674 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mrbsn" podStartSLOduration=15.796626843 podStartE2EDuration="39.181658019s" totalImagesPullingTime="23.385031176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:13.179678537 +0000 UTC m=+39.770438764" watchObservedRunningTime="2026-07-09 19:40:13.181658019 +0000 UTC m=+39.772418223" Jul 09 19:40:13.196477 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.196435 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-njtmb" podStartSLOduration=31.268935543 podStartE2EDuration="33.196421311s" totalImagesPullingTime="1.927485768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:13.195854372 +0000 UTC m=+39.786614598" watchObservedRunningTime="2026-07-09 19:40:13.196421311 +0000 UTC m=+39.787181494" Jul 09 19:40:13.312952 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.312854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:13.376204 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:13.376167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.149512 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.149320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.250022 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.249988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.624862 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.624748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.690402 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.690375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.755516 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.755489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.893106 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.893026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:14.993456 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:14.993423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:16.352624 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:16.352572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:17.087468 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:17.087430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:17.087657 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:17.087498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:17.087657 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:17.087526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:17.087657 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087636 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087656 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087660 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087656 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087711 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:25.087697045 +0000 UTC m=+51.678457228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087756 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:25.087739635 +0000 UTC m=+51.678499824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:17.087827 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:17.087782 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:25.087771806 +0000 UTC m=+51.678531995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:18.174060 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.174019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" event={"ID":"e0b5ada2-f474-4154-98c0-fbd8db86205e","Type":"ContainerStarted","Data":"cbc2a16b1766a03e86b751c693af09e58636cab459802fc62bfa97130e3fce55"} Jul 09 19:40:18.176156 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.176133 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/0.log" Jul 09 19:40:18.176257 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.176176 2562 generic.go:356] "Generic (PLEG): container finished" podID="38e13fea-c6f3-4bec-bccd-9a976004b5bc" containerID="470259689f61d549861ac47ab2ffc2b68df88aa9fbcb6fdde090dbacca2bc770" exitCode=255 Jul 09 19:40:18.176257 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.176206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-svwdq" event={"ID":"38e13fea-c6f3-4bec-bccd-9a976004b5bc","Type":"ContainerDied","Data":"470259689f61d549861ac47ab2ffc2b68df88aa9fbcb6fdde090dbacca2bc770"} Jul 09 19:40:18.176457 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.176439 2562 scope.go:117] "RemoveContainer" containerID="470259689f61d549861ac47ab2ffc2b68df88aa9fbcb6fdde090dbacca2bc770" Jul 09 19:40:18.178058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.178026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" event={"ID":"d488f561-09b6-48a7-af35-5805348a74f9","Type":"ContainerStarted","Data":"51c73b99c418cd7dd758f8dcc275ab07d84218cbd3bd932abf75f5c8e2176da2"} Jul 09 19:40:18.180039 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.180016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6qr9" event={"ID":"71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2","Type":"ContainerStarted","Data":"065d59b31bc578bd0aeeac65b4540e744e5b4146e743d4a9afe608c9ad25c6ac"} Jul 09 19:40:18.181457 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.181434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" event={"ID":"c9ee5626-3a98-4d0d-9667-fd294bea5c26","Type":"ContainerStarted","Data":"8bfaa13468f0fe41ff75af75812d47130a20bddbb82b2b5212b0d5ccd07671ff"} Jul 09 19:40:18.193545 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.193500 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zfmhn" podStartSLOduration=21.2893049 podStartE2EDuration="28.193483702s" totalImagesPullingTime="6.904178802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:18.192341706 +0000 UTC m=+44.783101927" watchObservedRunningTime="2026-07-09 19:40:18.193483702 +0000 UTC m=+44.784243908" Jul 09 19:40:18.210831 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.210794 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" podStartSLOduration=31.31774742 podStartE2EDuration="38.210779019s" totalImagesPullingTime="6.893031599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:18.21075331 +0000 UTC m=+44.801513516" watchObservedRunningTime="2026-07-09 19:40:18.210779019 +0000 UTC m=+44.801539225" Jul 09 19:40:18.230006 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.229966 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l6qr9" podStartSLOduration=2.329973905 podStartE2EDuration="9.229952967s" totalImagesPullingTime="6.899979062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:18.228931748 +0000 UTC m=+44.819691956" watchObservedRunningTime="2026-07-09 19:40:18.229952967 +0000 UTC m=+44.820713172" Jul 09 19:40:18.266903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.266853 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" podStartSLOduration=31.356312253 podStartE2EDuration="38.266832537s" totalImagesPullingTime="6.910520284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:18.266284894 +0000 UTC m=+44.857045100" watchObservedRunningTime="2026-07-09 19:40:18.266832537 +0000 UTC m=+44.857592743" Jul 09 19:40:18.573973 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.573857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:18.632083 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.632043 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:18.840431 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:18.840345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.032774 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.032734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.131993 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.131908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.185936 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.185909 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:40:19.186346 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.186334 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/0.log" Jul 09 19:40:19.186422 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.186367 2562 generic.go:356] "Generic (PLEG): container finished" podID="38e13fea-c6f3-4bec-bccd-9a976004b5bc" containerID="5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121" exitCode=255 Jul 09 19:40:19.186477 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.186450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-svwdq" event={"ID":"38e13fea-c6f3-4bec-bccd-9a976004b5bc","Type":"ContainerDied","Data":"5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121"} Jul 09 19:40:19.186537 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.186520 2562 scope.go:117] "RemoveContainer" containerID="470259689f61d549861ac47ab2ffc2b68df88aa9fbcb6fdde090dbacca2bc770" Jul 09 19:40:19.187000 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.186975 2562 scope.go:117] "RemoveContainer" containerID="5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121" Jul 09 19:40:19.187199 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:19.187171 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-svwdq_openshift-console-operator(38e13fea-c6f3-4bec-bccd-9a976004b5bc)\"" pod="openshift-console-operator/console-operator-7866497494-svwdq" podUID="38e13fea-c6f3-4bec-bccd-9a976004b5bc" Jul 09 19:40:19.279003 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.278961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.384744 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.384657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.551406 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.551373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:19.640909 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.640876 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:19.641059 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.640966 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:19.660550 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:19.660519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:20.047335 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:20.047246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:20.121549 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:20.121511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:20.193693 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:20.193668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:40:20.194100 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:20.193999 2562 scope.go:117] "RemoveContainer" containerID="5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121" Jul 09 19:40:20.194162 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:20.194144 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-svwdq_openshift-console-operator(38e13fea-c6f3-4bec-bccd-9a976004b5bc)\"" pod="openshift-console-operator/console-operator-7866497494-svwdq" podUID="38e13fea-c6f3-4bec-bccd-9a976004b5bc" Jul 09 19:40:21.182558 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:21.182522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:21.196865 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:21.196831 2562 scope.go:117] "RemoveContainer" containerID="5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121" Jul 09 19:40:21.197301 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:21.197041 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-svwdq_openshift-console-operator(38e13fea-c6f3-4bec-bccd-9a976004b5bc)\"" pod="openshift-console-operator/console-operator-7866497494-svwdq" podUID="38e13fea-c6f3-4bec-bccd-9a976004b5bc" Jul 09 19:40:21.248936 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:21.248901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:22.193749 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:22.193698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:24.174044 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:24.174004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:24.274970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:24.274933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:25.147054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:25.147013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:25.147071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147159 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147164 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147181 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b4dbc9c89-ztsqg: secret "image-registry-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:25.147190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147204 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls podName:e4869655-d404-4b5c-99b4-9131c36232a9 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:41.147191761 +0000 UTC m=+67.737951943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls") pod "cluster-samples-operator-65db858894-rw24s" (UID: "e4869655-d404-4b5c-99b4-9131c36232a9") : secret "samples-operator-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147220 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls podName:c86d9947-9e33-4824-bc5a-88e62f809860 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:41.147213498 +0000 UTC m=+67.737973680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls") pod "image-registry-b4dbc9c89-ztsqg" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860") : secret "image-registry-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147257 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:40:25.147316 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:25.147297 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls podName:98146308-3725-4523-80ec-7d626c3aee7d nodeName:}" failed. No retries permitted until 2026-07-09 19:40:41.147285561 +0000 UTC m=+67.738045745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls") pod "dns-default-w56d7" (UID: "98146308-3725-4523-80ec-7d626c3aee7d") : secret "dns-default-metrics-tls" not found Jul 09 19:40:25.716731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:25.716694 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:25.862432 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:25.862395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:26.029798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:26.029706 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:26.129805 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:26.129754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:26.231791 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:26.231755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:26.302161 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:26.302075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:26.580392 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:26.580306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:27.554817 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:27.554783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:27.954264 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:27.954235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:28.157035 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:28.156991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:28.351416 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:28.351318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:28.556809 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:28.556777 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:28.633288 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:28.633197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:29.568945 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:29.568907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:29.629825 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:29.629794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:29.754198 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:29.754164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:29.951774 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:29.951746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:30.149432 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:30.149395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:30.351388 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:30.351283 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:30.452878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:30.452844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:30.554282 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:30.554247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:31.347610 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:31.347550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:31.949849 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:31.949812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:32.148000 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:32.147965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:32.205565 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:32.205483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:32.355862 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:32.355827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:32.562650 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:32.562554 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:33.588552 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:33.588513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:33.772814 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:33.772782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:34.099816 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:34.099752 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8f5v" Jul 09 19:40:34.161550 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:34.161515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:34.353153 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:34.353059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:34.481270 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:34.481230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:34.751040 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:34.751005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:35.353412 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:35.353379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:35.724522 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:35.724490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:35.793424 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:35.793381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:35.948038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:35.948011 2562 scope.go:117] "RemoveContainer" containerID="5c007d48d7a90fba4310b5941cda1ed7163e281900e192d4690cebe9c7ff3121" Jul 09 19:40:35.955109 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:35.955078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:36.173196 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.173150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:36.243396 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.243320 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:40:36.243538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.243406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-svwdq" event={"ID":"38e13fea-c6f3-4bec-bccd-9a976004b5bc","Type":"ContainerStarted","Data":"543f28b50f82da31531be1e415a2fe7b87fd8ff84349d76aa8f4756d10d9089e"} Jul 09 19:40:36.243732 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.243713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:36.263139 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.263083 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-svwdq" podStartSLOduration=49.370025808 podStartE2EDuration="56.263067775s" totalImagesPullingTime="6.893041967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:36.261996642 +0000 UTC m=+62.852756870" watchObservedRunningTime="2026-07-09 19:40:36.263067775 +0000 UTC m=+62.853827980" Jul 09 19:40:36.439201 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.439166 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-svwdq" Jul 09 19:40:36.620126 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.620031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:36.720792 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.720760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:36.924494 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:36.924464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:37.298309 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:37.298218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:37.386803 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:37.386767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:37.500488 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:37.500452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:37.704093 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:37.704056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:37.760516 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:37.760486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:38.307640 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:38.307603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:38.514481 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:38.514440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.015098 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.015061 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.155560 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.155532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.211899 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.211866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.331071 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.330986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.392791 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.392760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:39.765411 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.765375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:39.765620 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.765472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:39.768510 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.768488 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 19:40:39.769191 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.769174 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 19:40:39.780280 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.780248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8afa780b-2966-4674-aa12-fe1b7b6e4a56-original-pull-secret\") pod \"global-pull-secret-syncer-9kgj4\" (UID: \"8afa780b-2966-4674-aa12-fe1b7b6e4a56\") " pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:39.780387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.780282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035c42e-61d8-4047-86c7-8018eaecc828-metrics-certs\") pod \"network-metrics-daemon-glnk7\" (UID: \"4035c42e-61d8-4047-86c7-8018eaecc828\") " pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:39.866845 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.866809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:39.869398 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.869363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fmbj\" (UniqueName: \"kubernetes.io/projected/91030579-d0bd-4adf-9004-dc7810e04abb-kube-api-access-4fmbj\") pod \"network-check-target-4xxq2\" (UID: \"91030579-d0bd-4adf-9004-dc7810e04abb\") " pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:39.959489 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.959461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kgj4" Jul 09 19:40:39.968695 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.968669 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mczbz\"" Jul 09 19:40:39.973831 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.973810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-95dhd\"" Jul 09 19:40:39.975874 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.975859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glnk7" Jul 09 19:40:39.982060 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:39.982047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:40.138306 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.138262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kgj4"] Jul 09 19:40:40.143811 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:40.143768 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afa780b_2966_4674_aa12_fe1b7b6e4a56.slice/crio-3668161c3a39974e42f9213271d104ec0cbc78284317e852312b11d12f940984 WatchSource:0}: Error finding container 3668161c3a39974e42f9213271d104ec0cbc78284317e852312b11d12f940984: Status 404 returned error can't find the container with id 3668161c3a39974e42f9213271d104ec0cbc78284317e852312b11d12f940984 Jul 09 19:40:40.157541 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.157519 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-glnk7"] Jul 09 19:40:40.160851 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:40.160821 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4035c42e_61d8_4047_86c7_8018eaecc828.slice/crio-e1cd0f03eeb8c6959fb886a6475b38ce9896762dab5bf1ef65b059e29b84a046 WatchSource:0}: Error finding container e1cd0f03eeb8c6959fb886a6475b38ce9896762dab5bf1ef65b059e29b84a046: Status 404 returned error can't find the container with id e1cd0f03eeb8c6959fb886a6475b38ce9896762dab5bf1ef65b059e29b84a046 Jul 09 19:40:40.182341 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.182319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xxq2"] Jul 09 19:40:40.184917 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:40.184895 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91030579_d0bd_4adf_9004_dc7810e04abb.slice/crio-212ecaf86e5b92a838315f2936a97d471bf2d997e9fec784f971540158038a6b WatchSource:0}: Error finding container 212ecaf86e5b92a838315f2936a97d471bf2d997e9fec784f971540158038a6b: Status 404 returned error can't find the container with id 212ecaf86e5b92a838315f2936a97d471bf2d997e9fec784f971540158038a6b Jul 09 19:40:40.257642 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.257607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glnk7" event={"ID":"4035c42e-61d8-4047-86c7-8018eaecc828","Type":"ContainerStarted","Data":"e1cd0f03eeb8c6959fb886a6475b38ce9896762dab5bf1ef65b059e29b84a046"} Jul 09 19:40:40.258618 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.258569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xxq2" event={"ID":"91030579-d0bd-4adf-9004-dc7810e04abb","Type":"ContainerStarted","Data":"212ecaf86e5b92a838315f2936a97d471bf2d997e9fec784f971540158038a6b"} Jul 09 19:40:40.259491 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:40.259472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kgj4" event={"ID":"8afa780b-2966-4674-aa12-fe1b7b6e4a56","Type":"ContainerStarted","Data":"3668161c3a39974e42f9213271d104ec0cbc78284317e852312b11d12f940984"} Jul 09 19:40:41.124731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.124697 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw"] Jul 09 19:40:41.146200 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.146174 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.149123 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.149083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw"] Jul 09 19:40:41.150388 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.150212 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 19:40:41.150388 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.150228 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 19:40:41.150542 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.150504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 19:40:41.150766 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.150750 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 19:40:41.151555 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.151537 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 19:40:41.151829 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.151816 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 19:40:41.152469 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.152432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/126af24a-e16c-49bb-a258-190dabf951c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177500 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5x5\" (UniqueName: \"kubernetes.io/projected/126af24a-e16c-49bb-a258-190dabf951c1-kube-api-access-5l5x5\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.177942 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:41.177942 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.177540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.180545 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.180520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"image-registry-b4dbc9c89-ztsqg\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:41.180878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.180858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98146308-3725-4523-80ec-7d626c3aee7d-metrics-tls\") pod \"dns-default-w56d7\" (UID: \"98146308-3725-4523-80ec-7d626c3aee7d\") " pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:41.181124 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.181041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4869655-d404-4b5c-99b4-9131c36232a9-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rw24s\" (UID: \"e4869655-d404-4b5c-99b4-9131c36232a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:41.266394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.266354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xxq2" event={"ID":"91030579-d0bd-4adf-9004-dc7810e04abb","Type":"ContainerStarted","Data":"a68e743d533121c34c0b6d20db1e2eda2f6fb12540c93751006b1d24116526ac"} Jul 09 19:40:41.266605 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.266524 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:40:41.278203 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.278376 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.280027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5x5\" (UniqueName: \"kubernetes.io/projected/126af24a-e16c-49bb-a258-190dabf951c1-kube-api-access-5l5x5\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.280027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.280027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.280027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.278760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/126af24a-e16c-49bb-a258-190dabf951c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.280027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.279657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/126af24a-e16c-49bb-a258-190dabf951c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.281325 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.281298 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.281426 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.281416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.281485 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.281443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.282364 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.282342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/126af24a-e16c-49bb-a258-190dabf951c1-ca\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.303899 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.302359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.309976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.309952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5x5\" (UniqueName: \"kubernetes.io/projected/126af24a-e16c-49bb-a258-190dabf951c1-kube-api-access-5l5x5\") pod \"cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw\" (UID: \"126af24a-e16c-49bb-a258-190dabf951c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.395994 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.395960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.408544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.408514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpggp\"" Jul 09 19:40:41.416416 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.416385 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:41.451406 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.451384 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nz9x5\"" Jul 09 19:40:41.459373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.459351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" Jul 09 19:40:41.469015 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.468981 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" Jul 09 19:40:41.469467 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.469443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtfn7\"" Jul 09 19:40:41.476902 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.476876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:41.491062 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.491019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.613003 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.612922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.623652 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.623596 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4xxq2" podStartSLOduration=67.62355756 podStartE2EDuration="1m7.62355756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:41.306503986 +0000 UTC m=+67.897264210" watchObservedRunningTime="2026-07-09 19:40:41.62355756 +0000 UTC m=+68.214317766" Jul 09 19:40:41.625403 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.625377 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:40:41.665810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.665731 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s"] Jul 09 19:40:41.683773 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.683746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw"] Jul 09 19:40:41.710357 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.710308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.713676 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:41.713651 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86d9947_9e33_4824_bc5a_88e62f809860.slice/crio-6c387a785cc829fd6f89532bd851038d56d13919f61eb1a5468c3d5a9b0b4433 WatchSource:0}: Error finding container 6c387a785cc829fd6f89532bd851038d56d13919f61eb1a5468c3d5a9b0b4433: Status 404 returned error can't find the container with id 6c387a785cc829fd6f89532bd851038d56d13919f61eb1a5468c3d5a9b0b4433 Jul 09 19:40:41.715431 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:41.715400 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod126af24a_e16c_49bb_a258_190dabf951c1.slice/crio-c576d5e79fe12cd6fb9eef340e769f405794041a8d484b3595f7260b941d7893 WatchSource:0}: Error finding container c576d5e79fe12cd6fb9eef340e769f405794041a8d484b3595f7260b941d7893: Status 404 returned error can't find the container with id c576d5e79fe12cd6fb9eef340e769f405794041a8d484b3595f7260b941d7893 Jul 09 19:40:41.809823 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.809769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:41.900203 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:41.900176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w56d7"] Jul 09 19:40:41.910188 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:40:41.910147 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98146308_3725_4523_80ec_7d626c3aee7d.slice/crio-77ce19d2c420ba13d6bfa2fc91fd8d43994db2fb79e19030d2e9cfd2fab51e20 WatchSource:0}: Error finding container 77ce19d2c420ba13d6bfa2fc91fd8d43994db2fb79e19030d2e9cfd2fab51e20: Status 404 returned error can't find the container with id 77ce19d2c420ba13d6bfa2fc91fd8d43994db2fb79e19030d2e9cfd2fab51e20 Jul 09 19:40:42.273007 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.272916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w56d7" event={"ID":"98146308-3725-4523-80ec-7d626c3aee7d","Type":"ContainerStarted","Data":"77ce19d2c420ba13d6bfa2fc91fd8d43994db2fb79e19030d2e9cfd2fab51e20"} Jul 09 19:40:42.274283 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.274245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" event={"ID":"e4869655-d404-4b5c-99b4-9131c36232a9","Type":"ContainerStarted","Data":"4a9879021df48e8742b99061cb42e45888efad82a0d5d029e42256cf46d0a73c"} Jul 09 19:40:42.276236 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.276212 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glnk7" event={"ID":"4035c42e-61d8-4047-86c7-8018eaecc828","Type":"ContainerStarted","Data":"4613cd76d7441e3e888e35df10a09feb77c8904302046aeb030412f1241f5d2e"} Jul 09 19:40:42.276363 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.276257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glnk7" event={"ID":"4035c42e-61d8-4047-86c7-8018eaecc828","Type":"ContainerStarted","Data":"0e94477cb1931d3beb746be11567b3efd92b09df316e1f51d7e0430bcc61972f"} Jul 09 19:40:42.277482 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.277462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" event={"ID":"126af24a-e16c-49bb-a258-190dabf951c1","Type":"ContainerStarted","Data":"c576d5e79fe12cd6fb9eef340e769f405794041a8d484b3595f7260b941d7893"} Jul 09 19:40:42.278909 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.278889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" event={"ID":"c86d9947-9e33-4824-bc5a-88e62f809860","Type":"ContainerStarted","Data":"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7"} Jul 09 19:40:42.279066 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.279050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:40:42.279137 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.279120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" event={"ID":"c86d9947-9e33-4824-bc5a-88e62f809860","Type":"ContainerStarted","Data":"6c387a785cc829fd6f89532bd851038d56d13919f61eb1a5468c3d5a9b0b4433"} Jul 09 19:40:42.294753 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.294711 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-glnk7" podStartSLOduration=66.703613508 podStartE2EDuration="1m8.294698429s" totalImagesPullingTime="1.591084921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:42.293734326 +0000 UTC m=+68.884494552" watchObservedRunningTime="2026-07-09 19:40:42.294698429 +0000 UTC m=+68.885458634" Jul 09 19:40:42.315955 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.315911 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" podStartSLOduration=70.315899063 podStartE2EDuration="1m10.315899063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:42.314719389 +0000 UTC m=+68.905479614" watchObservedRunningTime="2026-07-09 19:40:42.315899063 +0000 UTC m=+68.906659288" Jul 09 19:40:42.416054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.416017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:42.601736 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.601653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:42.672189 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:42.672155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:45.341115 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:45.341067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:45.414890 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:45.414828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:47.751266 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:47.751226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:47.815394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:47.815359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:48.303381 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:48.303346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kgj4" event={"ID":"8afa780b-2966-4674-aa12-fe1b7b6e4a56","Type":"ContainerStarted","Data":"b534dc4346e97db85f111aaa490fb2c85767f8043cd3f27aae0c4ff2210167bc"} Jul 09 19:40:48.304788 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:48.304766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w56d7" event={"ID":"98146308-3725-4523-80ec-7d626c3aee7d","Type":"ContainerStarted","Data":"fded5834aa87b2f90fe30706e8d5e2357b197eb238227649bfebd93bc98ca057"} Jul 09 19:40:48.306105 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:48.306083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" event={"ID":"e4869655-d404-4b5c-99b4-9131c36232a9","Type":"ContainerStarted","Data":"d54ff78c2f4c4044a2dd3b5c28ea3e3f0557680bba4cf601eb5ee69fd62c01ed"} Jul 09 19:40:48.320158 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:48.320115 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9kgj4" podStartSLOduration=66.542106972 podStartE2EDuration="1m14.320100796s" totalImagesPullingTime="7.777993824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:48.319548266 +0000 UTC m=+74.910308494" watchObservedRunningTime="2026-07-09 19:40:48.320100796 +0000 UTC m=+74.910860991" Jul 09 19:40:49.168539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.168491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:49.248998 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.248959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:49.314246 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.314214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w56d7" event={"ID":"98146308-3725-4523-80ec-7d626c3aee7d","Type":"ContainerStarted","Data":"c298d9f0f64e941bc0ee8c4a7e415630127c3c04d59413c582f7afb2119846c9"} Jul 09 19:40:49.314421 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.314317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:49.316218 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.316188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" event={"ID":"e4869655-d404-4b5c-99b4-9131c36232a9","Type":"ContainerStarted","Data":"704c953b88d467f78f9ded110ae8c8f12454e97f9731a5ce7ce4d13c1f5f62fe"} Jul 09 19:40:49.318365 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.318339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" event={"ID":"126af24a-e16c-49bb-a258-190dabf951c1","Type":"ContainerStarted","Data":"fa8683c75aed478987a0322c4092576c34cc8ddaf53f65af3fb3b6107ac8b590"} Jul 09 19:40:49.318459 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.318372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" event={"ID":"126af24a-e16c-49bb-a258-190dabf951c1","Type":"ContainerStarted","Data":"fa1f54ec34802afac50e8274df36d646fac8d322f04ac73adbc80c6564355294"} Jul 09 19:40:49.318459 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.318388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" event={"ID":"126af24a-e16c-49bb-a258-190dabf951c1","Type":"ContainerStarted","Data":"7193ca3acbaf6782325b45181bc17a0c4e63288598f6d7c2243fec9063d634e1"} Jul 09 19:40:49.332956 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.332919 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w56d7" podStartSLOduration=34.317020517 podStartE2EDuration="40.332905897s" totalImagesPullingTime="6.01588538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:49.331181741 +0000 UTC m=+75.921941963" watchObservedRunningTime="2026-07-09 19:40:49.332905897 +0000 UTC m=+75.923666101" Jul 09 19:40:49.339296 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.339270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:49.352658 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.352604 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rw24s" podStartSLOduration=63.215102602 podStartE2EDuration="1m9.352570741s" totalImagesPullingTime="6.137468139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:49.351802942 +0000 UTC m=+75.942563160" watchObservedRunningTime="2026-07-09 19:40:49.352570741 +0000 UTC m=+75.943330947" Jul 09 19:40:49.380534 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.380483 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8cc8c5fc7-wtvzw" podStartSLOduration=1.90368051 podStartE2EDuration="8.380467889s" totalImagesPullingTime="6.476787379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:40:49.378986425 +0000 UTC m=+75.969746630" watchObservedRunningTime="2026-07-09 19:40:49.380467889 +0000 UTC m=+75.971228094" Jul 09 19:40:49.421667 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.421571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:49.538237 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.538204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:49.980391 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:49.980352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.077494 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.077457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.166113 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.166086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.236072 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.235984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.600631 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.600527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.716507 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.716477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.741280 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.741248 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6zf6f"] Jul 09 19:40:50.762825 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.762802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zf6f"] Jul 09 19:40:50.762976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.762907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.765710 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.765684 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 19:40:50.765839 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.765718 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 19:40:50.765839 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.765684 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 19:40:50.766837 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.766819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hw6z8\"" Jul 09 19:40:50.766941 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.766842 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 19:40:50.859725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.859654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df40c511-b6f9-4288-a237-c13f84d5e539-data-volume\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.859725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.859711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.859912 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.859766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.859912 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.859808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df40c511-b6f9-4288-a237-c13f84d5e539-crio-socket\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.859912 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.859829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnv9\" (UniqueName: \"kubernetes.io/projected/df40c511-b6f9-4288-a237-c13f84d5e539-kube-api-access-6vnv9\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.887125 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.887094 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:50.960739 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.960892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.960892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df40c511-b6f9-4288-a237-c13f84d5e539-crio-socket\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.960892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnv9\" (UniqueName: \"kubernetes.io/projected/df40c511-b6f9-4288-a237-c13f84d5e539-kube-api-access-6vnv9\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.960892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df40c511-b6f9-4288-a237-c13f84d5e539-data-volume\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.960892 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:50.960860 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:40:50.961168 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.960933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df40c511-b6f9-4288-a237-c13f84d5e539-crio-socket\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.961168 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:50.960880 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:40:50.961168 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:50.960965 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:51.46094178 +0000 UTC m=+78.051701979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : secret "insights-runtime-extractor-tls" not found Jul 09 19:40:50.961168 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:50.961025 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:51.461008881 +0000 UTC m=+78.051769083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : configmap "kube-rbac-proxy" not found Jul 09 19:40:50.961300 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.961178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df40c511-b6f9-4288-a237-c13f84d5e539-data-volume\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:50.969604 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:50.969569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnv9\" (UniqueName: \"kubernetes.io/projected/df40c511-b6f9-4288-a237-c13f84d5e539-kube-api-access-6vnv9\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:51.427563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.427528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:51.465637 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.465602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:51.465805 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.465641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:51.465805 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:51.465727 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:40:51.465805 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:51.465765 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:40:51.465805 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:51.465781 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:52.465767036 +0000 UTC m=+79.056527219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : configmap "kube-rbac-proxy" not found Jul 09 19:40:51.465955 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:51.465824 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:52.465805874 +0000 UTC m=+79.056566061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : secret "insights-runtime-extractor-tls" not found Jul 09 19:40:51.527273 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.527244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:51.860651 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.860547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:51.958802 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:51.958764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:52.474893 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:52.474863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:52.475335 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:52.474901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:52.475335 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:52.474984 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:40:52.475335 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:52.475017 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:40:52.475335 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:52.475034 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:54.475021402 +0000 UTC m=+81.065781585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : configmap "kube-rbac-proxy" not found Jul 09 19:40:52.475335 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:52.475070 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:54.475057687 +0000 UTC m=+81.065817869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : secret "insights-runtime-extractor-tls" not found Jul 09 19:40:52.725156 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:52.725070 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:40:52.811615 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:52.811566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:52.927531 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:52.927503 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/2.log" Jul 09 19:40:53.206990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:53.206959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:53.273494 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:53.273453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:54.174852 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.174819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:54.251285 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.251254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:54.491041 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.490964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:54.491041 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.491004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:54.491227 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:54.491087 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:40:54.491227 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:54.491118 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:40:54.491227 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:54.491141 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:58.491128024 +0000 UTC m=+85.081888207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : configmap "kube-rbac-proxy" not found Jul 09 19:40:54.491227 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:54.491179 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:40:58.491156242 +0000 UTC m=+85.081916425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : secret "insights-runtime-extractor-tls" not found Jul 09 19:40:54.493843 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.493820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:54.526501 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.526476 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w56d7_98146308-3725-4523-80ec-7d626c3aee7d/dns/0.log" Jul 09 19:40:54.725938 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.725822 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w56d7_98146308-3725-4523-80ec-7d626c3aee7d/kube-rbac-proxy/0.log" Jul 09 19:40:54.927654 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:54.927630 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wtfh_2dd2e12b-47b3-42ed-8296-3ecea75e2d26/dns-node-resolver/0.log" Jul 09 19:40:55.345810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:55.345720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:55.445919 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:55.445847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:55.724501 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:55.724469 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b4dbc9c89-ztsqg_c86d9947-9e33-4824-bc5a-88e62f809860/registry/0.log" Jul 09 19:40:55.926629 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:55.926574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j984v_9e743b07-b293-4fb3-9e5e-982a16987054/node-ca/0.log" Jul 09 19:40:57.124570 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:57.124543 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6qr9_71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2/serve-healthcheck-canary/0.log" Jul 09 19:40:57.341343 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:57.341308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:57.527557 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:57.527503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:57.632743 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:57.632702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:58.519225 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:58.519177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:58.519225 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:58.519230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:40:58.519703 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:58.519330 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:40:58.519703 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:58.519333 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:40:58.519703 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:58.519402 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:41:06.519387146 +0000 UTC m=+93.110147333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : secret "insights-runtime-extractor-tls" not found Jul 09 19:40:58.519703 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:58.519418 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm podName:df40c511-b6f9-4288-a237-c13f84d5e539 nodeName:}" failed. No retries permitted until 2026-07-09 19:41:06.51941122 +0000 UTC m=+93.110171403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6zf6f" (UID: "df40c511-b6f9-4288-a237-c13f84d5e539") : configmap "kube-rbac-proxy" not found Jul 09 19:40:59.133286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.133253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.234038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.234009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.323833 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.323801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w56d7" Jul 09 19:40:59.334941 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.334913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.537398 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.537360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.609747 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.609720 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2nwl5"] Jul 09 19:40:59.613070 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.613051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.616363 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.616339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 19:40:59.616471 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.616365 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 19:40:59.617558 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.617541 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wcdj6\"" Jul 09 19:40:59.617746 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.617564 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 19:40:59.617746 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.617569 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 19:40:59.617746 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.617620 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 19:40:59.617746 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.617678 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 19:40:59.646932 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.646901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.728158 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-textfile\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft5k\" (UniqueName: \"kubernetes.io/projected/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-kube-api-access-bft5k\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728453 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728453 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-sys\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728453 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-root\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728620 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-metrics-client-ca\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.728620 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.728524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-wtmp\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.741191 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.741159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.829709 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-metrics-client-ca\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.829709 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-wtmp\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.829895 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-wtmp\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.829895 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-textfile\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.829895 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830012 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft5k\" (UniqueName: \"kubernetes.io/projected/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-kube-api-access-bft5k\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830012 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.829925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830094 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:59.830031 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 19:40:59.830094 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830193 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:40:59.830098 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls podName:6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf nodeName:}" failed. No retries permitted until 2026-07-09 19:41:00.330078475 +0000 UTC m=+86.920838665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls") pod "node-exporter-2nwl5" (UID: "6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf") : secret "node-exporter-tls" not found Jul 09 19:40:59.830193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-textfile\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-sys\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-root\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-metrics-client-ca\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-root\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830387 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-sys\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.830456 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.830395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.832367 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.832349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.845388 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.845362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:40:59.845944 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.845924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft5k\" (UniqueName: \"kubernetes.io/projected/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-kube-api-access-bft5k\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:40:59.969504 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:40:59.969467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.070053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.070003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.286657 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.286625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.336020 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.335991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:41:00.338407 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.338374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf-node-exporter-tls\") pod \"node-exporter-2nwl5\" (UID: \"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf\") " pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:41:00.475485 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.475446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.522369 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.522336 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nwl5" Jul 09 19:41:00.533488 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:41:00.533447 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d22e7b9_cf4a_4afa_8ca3_50bda477b3cf.slice/crio-7727e00bc1884ff99ec3b20061bda1c331d4d4b64726b34e6b192d90e775651e WatchSource:0}: Error finding container 7727e00bc1884ff99ec3b20061bda1c331d4d4b64726b34e6b192d90e775651e: Status 404 returned error can't find the container with id 7727e00bc1884ff99ec3b20061bda1c331d4d4b64726b34e6b192d90e775651e Jul 09 19:41:00.578692 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.578660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.680038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.680004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:00.893784 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:00.893749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:01.120030 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:01.119992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:01.364538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:01.364447 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nwl5" event={"ID":"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf","Type":"ContainerStarted","Data":"7727e00bc1884ff99ec3b20061bda1c331d4d4b64726b34e6b192d90e775651e"} Jul 09 19:41:01.422746 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:01.422701 2562 patch_prober.go:28] interesting pod/image-registry-b4dbc9c89-ztsqg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 19:41:01.422920 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:01.422779 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:41:02.369687 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.369651 2562 generic.go:356] "Generic (PLEG): container finished" podID="6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf" containerID="e315bf7da4efd200c8d5a314b705b4ed6fadb5a84e720a3179b6c1930b4e51cd" exitCode=0 Jul 09 19:41:02.370155 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.369750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nwl5" event={"ID":"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf","Type":"ContainerDied","Data":"e315bf7da4efd200c8d5a314b705b4ed6fadb5a84e720a3179b6c1930b4e51cd"} Jul 09 19:41:02.629089 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.628989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:02.694858 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.694826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:02.791003 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.790964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:02.874623 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:02.874564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:03.251843 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.251808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:03.289669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.289641 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:41:03.374572 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.374537 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nwl5" event={"ID":"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf","Type":"ContainerStarted","Data":"bc78a03f3c015bdf200cc3964fcd59101c8ebcb55215423e1ec092cfe0c9d979"} Jul 09 19:41:03.374941 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.374610 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nwl5" event={"ID":"6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf","Type":"ContainerStarted","Data":"06b9e360532bdbfca53809fd51e4f7177611e87ccb7f34174c1298389a481b00"} Jul 09 19:41:03.394142 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.394098 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2nwl5" podStartSLOduration=3.337278229 podStartE2EDuration="4.39408516s" totalImagesPullingTime="985.244934ms" totalInitContainerRuntime="71.561997ms" isStatefulPod=true podCreationTimestamp="2026-07-09 19:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:41:03.393314947 +0000 UTC m=+89.984075188" watchObservedRunningTime="2026-07-09 19:41:03.39408516 +0000 UTC m=+89.984845365" Jul 09 19:41:03.574856 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.574754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:03.672486 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:03.672449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.347990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.347942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.392606 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.392555 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:41:04.422617 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.422564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.515572 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.515536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.770768 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.770729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.865606 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.865549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:04.992958 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:04.992910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:06.025534 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.025503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:06.127442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.127398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:06.516445 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.516411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:06.586009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.585982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:41:06.586162 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.586051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:41:06.586489 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.586472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df40c511-b6f9-4288-a237-c13f84d5e539-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:41:06.588477 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.588456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df40c511-b6f9-4288-a237-c13f84d5e539-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6zf6f\" (UID: \"df40c511-b6f9-4288-a237-c13f84d5e539\") " pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:41:06.618634 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.618599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:06.672502 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.672476 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6zf6f" Jul 09 19:41:06.796024 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:06.795946 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6zf6f"] Jul 09 19:41:06.799134 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:41:06.799107 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf40c511_b6f9_4288_a237_c13f84d5e539.slice/crio-88a70a4d984c6cad35475b606bbf45692db1ca32e441bb7a8bb87c27097acc24 WatchSource:0}: Error finding container 88a70a4d984c6cad35475b606bbf45692db1ca32e441bb7a8bb87c27097acc24: Status 404 returned error can't find the container with id 88a70a4d984c6cad35475b606bbf45692db1ca32e441bb7a8bb87c27097acc24 Jul 09 19:41:07.391282 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:07.391228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zf6f" event={"ID":"df40c511-b6f9-4288-a237-c13f84d5e539","Type":"ContainerStarted","Data":"88a70a4d984c6cad35475b606bbf45692db1ca32e441bb7a8bb87c27097acc24"} Jul 09 19:41:08.945412 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:08.945367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:09.046358 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:09.046319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:09.246736 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:09.246624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:09.346394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:09.346348 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:10.258286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:10.258248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:10.350717 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:10.350677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:11.363443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:11.363412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:11.408690 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:11.408665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zf6f" event={"ID":"df40c511-b6f9-4288-a237-c13f84d5e539","Type":"ContainerStarted","Data":"50ec0d1651525504966d8e5682c40f808b78d56cbe123e96ea112cd42e9ac07f"} Jul 09 19:41:11.463717 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:11.463679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:12.281796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:12.281715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xxq2" Jul 09 19:41:12.415205 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:12.415156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zf6f" event={"ID":"df40c511-b6f9-4288-a237-c13f84d5e539","Type":"ContainerStarted","Data":"7ea4b399e1498dc30ba0fc977c9806093e12851e207d582a74d37e13af1574af"} Jul 09 19:41:14.161757 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:14.161698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:14.261394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:14.261341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:15.428622 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:15.428571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6zf6f" event={"ID":"df40c511-b6f9-4288-a237-c13f84d5e539","Type":"ContainerStarted","Data":"b988f6c6241f4bae55905b583dcce30e51132be83642401b20f82fbc8298c876"} Jul 09 19:41:15.455987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:15.455940 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6zf6f" podStartSLOduration=17.843426427 podStartE2EDuration="25.455923965s" totalImagesPullingTime="7.612497538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:40:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:41:15.454303698 +0000 UTC m=+102.045063924" watchObservedRunningTime="2026-07-09 19:41:15.455923965 +0000 UTC m=+102.046684170" Jul 09 19:41:15.525312 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:15.525271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:15.627078 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:15.627042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:20.440371 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:20.440333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:20.501845 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:20.501807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:21.057789 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:21.057758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:21.121136 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:21.121104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:21.184123 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:21.184089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:24.156133 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:24.156099 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:24.213401 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:24.213359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:28.474377 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:28.474342 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9ee5626-3a98-4d0d-9667-fd294bea5c26" containerID="8bfaa13468f0fe41ff75af75812d47130a20bddbb82b2b5212b0d5ccd07671ff" exitCode=0 Jul 09 19:41:28.474793 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:28.474405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" event={"ID":"c9ee5626-3a98-4d0d-9667-fd294bea5c26","Type":"ContainerDied","Data":"8bfaa13468f0fe41ff75af75812d47130a20bddbb82b2b5212b0d5ccd07671ff"} Jul 09 19:41:28.474793 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:28.474711 2562 scope.go:117] "RemoveContainer" containerID="8bfaa13468f0fe41ff75af75812d47130a20bddbb82b2b5212b0d5ccd07671ff" Jul 09 19:41:29.413160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.413095 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" containerName="registry" containerID="cri-o://04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7" gracePeriod=30 Jul 09 19:41:29.479144 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.479112 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-9zlrt" event={"ID":"c9ee5626-3a98-4d0d-9667-fd294bea5c26","Type":"ContainerStarted","Data":"e9b2681fdb4cb3d69a18175a80535c694198f5778c940bcd24e7b138ee71f705"} Jul 09 19:41:29.654271 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.654251 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:41:29.669337 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669286 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669417 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669336 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669417 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669369 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669417 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669404 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669536 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669429 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx9b9\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669536 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669457 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669536 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669504 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669753 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669714 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration\") pod \"c86d9947-9e33-4824-bc5a-88e62f809860\" (UID: \"c86d9947-9e33-4824-bc5a-88e62f809860\") " Jul 09 19:41:29.669835 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.669741 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:41:29.670021 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.670000 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:41:29.670145 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.670046 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-registry-certificates\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.671985 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.671944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:41:29.672226 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.672194 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:29.672436 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.672406 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:29.672539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.672515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:41:29.672704 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.672672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9" (OuterVolumeSpecName: "kube-api-access-wx9b9") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "kube-api-access-wx9b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:29.679964 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.679940 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c86d9947-9e33-4824-bc5a-88e62f809860" (UID: "c86d9947-9e33-4824-bc5a-88e62f809860"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:41:29.770892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770860 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c86d9947-9e33-4824-bc5a-88e62f809860-trusted-ca\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.770892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770889 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-image-registry-private-configuration\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.770892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770900 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c86d9947-9e33-4824-bc5a-88e62f809860-installation-pull-secrets\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.771102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770908 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-registry-tls\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.771102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770917 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c86d9947-9e33-4824-bc5a-88e62f809860-ca-trust-extracted\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.771102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770926 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx9b9\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-kube-api-access-wx9b9\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:29.771102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:29.770935 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86d9947-9e33-4824-bc5a-88e62f809860-bound-sa-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:41:30.483680 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.483650 2562 generic.go:356] "Generic (PLEG): container finished" podID="c86d9947-9e33-4824-bc5a-88e62f809860" containerID="04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7" exitCode=0 Jul 09 19:41:30.484085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.483690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" event={"ID":"c86d9947-9e33-4824-bc5a-88e62f809860","Type":"ContainerDied","Data":"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7"} Jul 09 19:41:30.484085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.483702 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" Jul 09 19:41:30.484085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.483728 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4dbc9c89-ztsqg" event={"ID":"c86d9947-9e33-4824-bc5a-88e62f809860","Type":"ContainerDied","Data":"6c387a785cc829fd6f89532bd851038d56d13919f61eb1a5468c3d5a9b0b4433"} Jul 09 19:41:30.484085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.483747 2562 scope.go:117] "RemoveContainer" containerID="04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7" Jul 09 19:41:30.491439 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.491422 2562 scope.go:117] "RemoveContainer" containerID="04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7" Jul 09 19:41:30.491727 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:41:30.491703 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7\": container with ID starting with 04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7 not found: ID does not exist" containerID="04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7" Jul 09 19:41:30.491786 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.491736 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7"} err="failed to get container status \"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7\": rpc error: code = NotFound desc = could not find container \"04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7\": container with ID starting with 04b76168235d4c9ae5073870428b4b02b820b78be3d5e3adeb832fca06779bb7 not found: ID does not exist" Jul 09 19:41:30.501477 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.501455 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:41:30.505885 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:30.505863 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b4dbc9c89-ztsqg"] Jul 09 19:41:31.951827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:31.951795 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" path="/var/lib/kubelet/pods/c86d9947-9e33-4824-bc5a-88e62f809860/volumes" Jul 09 19:41:33.496518 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:33.496480 2562 generic.go:356] "Generic (PLEG): container finished" podID="d488f561-09b6-48a7-af35-5805348a74f9" containerID="51c73b99c418cd7dd758f8dcc275ab07d84218cbd3bd932abf75f5c8e2176da2" exitCode=0 Jul 09 19:41:33.496933 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:33.496555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" event={"ID":"d488f561-09b6-48a7-af35-5805348a74f9","Type":"ContainerDied","Data":"51c73b99c418cd7dd758f8dcc275ab07d84218cbd3bd932abf75f5c8e2176da2"} Jul 09 19:41:33.496933 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:33.496881 2562 scope.go:117] "RemoveContainer" containerID="51c73b99c418cd7dd758f8dcc275ab07d84218cbd3bd932abf75f5c8e2176da2" Jul 09 19:41:34.501902 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:34.501867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ch6g8" event={"ID":"d488f561-09b6-48a7-af35-5805348a74f9","Type":"ContainerStarted","Data":"09d2c10a6f203ea7c82599f9497a486d4dca26d2e8acb28eec3f208d65a3abb9"} Jul 09 19:41:36.768345 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:36.768312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:39.159903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:39.159864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:39.213030 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:39.212999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:40.799090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:40.799057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:47.397824 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:47.397789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:47.496768 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:47.496731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:47.597838 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:47.597800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:41:50.656154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:41:50.656121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:15.757493 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:15.757447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:15.856421 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:15.856384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:17.018501 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:17.018458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:17.084533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:17.084497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:21.035035 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:21.035001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:21.098135 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:21.098100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:27.672484 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:27.672394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:27.727643 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:27.727618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:28.285376 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:28.285340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:28.386493 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:28.386461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:28.442998 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:28.442971 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:39.161278 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:39.161245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:39.266591 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:39.266553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:39.345178 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:39.345148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:42:58.945372 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:42:58.945333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:02.944053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:02.944015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:03.064835 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.064801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:03.804696 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.804665 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg"] Jul 09 19:43:03.804930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.804919 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" containerName="registry" Jul 09 19:43:03.804972 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.804932 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" containerName="registry" Jul 09 19:43:03.805010 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.804973 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86d9947-9e33-4824-bc5a-88e62f809860" containerName="registry" Jul 09 19:43:03.807947 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.807932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.810701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.810675 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:43:03.810701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.810697 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jhwl5\"" Jul 09 19:43:03.811757 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.811735 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:43:03.816305 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.816286 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg"] Jul 09 19:43:03.833146 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.833123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5qm\" (UniqueName: \"kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.833258 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.833163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.833258 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.833204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.933690 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.933659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.933873 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.933704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.933873 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.933763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5qm\" (UniqueName: \"kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.934127 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.934108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.934191 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.934131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:03.943976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:03.943951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5qm\" (UniqueName: \"kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:04.117539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:04.117461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:04.245116 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:04.245088 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg"] Jul 09 19:43:04.247868 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:43:04.247843 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16753cc9_101d_42fb_91e5_c959ec5d7d6c.slice/crio-7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c WatchSource:0}: Error finding container 7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c: Status 404 returned error can't find the container with id 7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c Jul 09 19:43:04.809633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:04.809599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerStarted","Data":"7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c"} Jul 09 19:43:11.838046 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:11.838009 2562 generic.go:356] "Generic (PLEG): container finished" podID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerID="4d1ae113af872207b6dc016e968ec10c6dff855305f8c52ad3470470d62df34a" exitCode=0 Jul 09 19:43:11.838426 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:11.838083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerDied","Data":"4d1ae113af872207b6dc016e968ec10c6dff855305f8c52ad3470470d62df34a"} Jul 09 19:43:14.852442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:14.852412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerStarted","Data":"b736fa7571bd00eb3a3ea6394413077bf6cbad2b6771a545a39558bcea52a79e"} Jul 09 19:43:15.857651 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:15.857619 2562 generic.go:356] "Generic (PLEG): container finished" podID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerID="b736fa7571bd00eb3a3ea6394413077bf6cbad2b6771a545a39558bcea52a79e" exitCode=0 Jul 09 19:43:15.858015 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:15.857671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerDied","Data":"b736fa7571bd00eb3a3ea6394413077bf6cbad2b6771a545a39558bcea52a79e"} Jul 09 19:43:23.889847 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:23.889815 2562 generic.go:356] "Generic (PLEG): container finished" podID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerID="ea73d7c9f0a8e95d65d09bcc30dad24929400b7b20e57ffca1b9f289f9e67f99" exitCode=0 Jul 09 19:43:23.890208 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:23.889901 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerDied","Data":"ea73d7c9f0a8e95d65d09bcc30dad24929400b7b20e57ffca1b9f289f9e67f99"} Jul 09 19:43:25.010302 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.010280 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:25.103484 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.103454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle\") pod \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " Jul 09 19:43:25.103701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.103491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p5qm\" (UniqueName: \"kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm\") pod \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " Jul 09 19:43:25.103701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.103519 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util\") pod \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\" (UID: \"16753cc9-101d-42fb-91e5-c959ec5d7d6c\") " Jul 09 19:43:25.104140 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.104115 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle" (OuterVolumeSpecName: "bundle") pod "16753cc9-101d-42fb-91e5-c959ec5d7d6c" (UID: "16753cc9-101d-42fb-91e5-c959ec5d7d6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:43:25.105770 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.105745 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm" (OuterVolumeSpecName: "kube-api-access-2p5qm") pod "16753cc9-101d-42fb-91e5-c959ec5d7d6c" (UID: "16753cc9-101d-42fb-91e5-c959ec5d7d6c"). InnerVolumeSpecName "kube-api-access-2p5qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:43:25.108465 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.108441 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util" (OuterVolumeSpecName: "util") pod "16753cc9-101d-42fb-91e5-c959ec5d7d6c" (UID: "16753cc9-101d-42fb-91e5-c959ec5d7d6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:43:25.204466 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.204358 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:25.204466 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.204407 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5qm\" (UniqueName: \"kubernetes.io/projected/16753cc9-101d-42fb-91e5-c959ec5d7d6c-kube-api-access-2p5qm\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:25.204466 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.204421 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16753cc9-101d-42fb-91e5-c959ec5d7d6c-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:25.898495 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.898459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" event={"ID":"16753cc9-101d-42fb-91e5-c959ec5d7d6c","Type":"ContainerDied","Data":"7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c"} Jul 09 19:43:25.898495 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.898474 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540df2jg" Jul 09 19:43:25.898495 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:25.898504 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bffe5629067475b9e5af4758caefb5a3f34f28d259651ae8537bcc38aab856c" Jul 09 19:43:29.856364 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:29.856325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:30.663538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:30.663494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:31.778220 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778189 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7"] Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778417 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="util" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778428 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="util" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778439 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="pull" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778444 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="pull" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778459 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="extract" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778464 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="extract" Jul 09 19:43:31.778607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.778507 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="16753cc9-101d-42fb-91e5-c959ec5d7d6c" containerName="extract" Jul 09 19:43:31.781470 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.781455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.784401 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.784381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kp5gj\"" Jul 09 19:43:31.784825 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.784804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 19:43:31.785489 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.785473 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:43:31.796689 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.796670 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7"] Jul 09 19:43:31.851593 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.851563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733780ad-818f-49f7-8812-236d0c3bac35-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.851718 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.851648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6dw\" (UniqueName: \"kubernetes.io/projected/733780ad-818f-49f7-8812-236d0c3bac35-kube-api-access-7l6dw\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.952206 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.952182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6dw\" (UniqueName: \"kubernetes.io/projected/733780ad-818f-49f7-8812-236d0c3bac35-kube-api-access-7l6dw\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.952345 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.952225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733780ad-818f-49f7-8812-236d0c3bac35-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.952552 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.952536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733780ad-818f-49f7-8812-236d0c3bac35-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:31.961301 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:31.961276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6dw\" (UniqueName: \"kubernetes.io/projected/733780ad-818f-49f7-8812-236d0c3bac35-kube-api-access-7l6dw\") pod \"cert-manager-operator-controller-manager-55c444bf86-qsqs7\" (UID: \"733780ad-818f-49f7-8812-236d0c3bac35\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:32.089735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:32.089667 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" Jul 09 19:43:32.221538 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:32.221508 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7"] Jul 09 19:43:32.225552 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:43:32.225529 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733780ad_818f_49f7_8812_236d0c3bac35.slice/crio-e1c1861d00cf84715fb2b40849aec431e966a030a6495ed7be8930859ae519d0 WatchSource:0}: Error finding container e1c1861d00cf84715fb2b40849aec431e966a030a6495ed7be8930859ae519d0: Status 404 returned error can't find the container with id e1c1861d00cf84715fb2b40849aec431e966a030a6495ed7be8930859ae519d0 Jul 09 19:43:32.925238 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:32.925106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" event={"ID":"733780ad-818f-49f7-8812-236d0c3bac35","Type":"ContainerStarted","Data":"e1c1861d00cf84715fb2b40849aec431e966a030a6495ed7be8930859ae519d0"} Jul 09 19:43:33.669283 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:33.669244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:33.768871 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:33.768838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:33.934519 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:33.934424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" event={"ID":"733780ad-818f-49f7-8812-236d0c3bac35","Type":"ContainerStarted","Data":"9afa55f0b06ab242e5ed0737654b18723757541c9873471584b3b8f43e06effa"} Jul 09 19:43:33.961458 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:33.961412 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qsqs7" podStartSLOduration=1.3576835520000001 podStartE2EDuration="2.961395003s" totalImagesPullingTime="1.603711451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:43:33.957902647 +0000 UTC m=+240.548662852" watchObservedRunningTime="2026-07-09 19:43:33.961395003 +0000 UTC m=+240.552155209" Jul 09 19:43:34.376692 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:34.376610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:34.482441 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:34.482410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:35.300170 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.300135 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr"] Jul 09 19:43:35.303364 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.303348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.306149 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.306121 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:43:35.306280 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.306156 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:43:35.306280 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.306210 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jhwl5\"" Jul 09 19:43:35.312531 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.312514 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr"] Jul 09 19:43:35.384633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.384608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.384633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.384635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.384808 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.384706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnjm\" (UniqueName: \"kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.485599 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.485550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.485752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.485616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.485752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.485681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnjm\" (UniqueName: \"kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.486015 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.485993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.486056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.486006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.495632 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.495604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnjm\" (UniqueName: \"kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.612940 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.612862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:35.750612 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.750415 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr"] Jul 09 19:43:35.753407 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:43:35.753380 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3d921d_eff0_492e_ba7e_69e32c2ed2d5.slice/crio-a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc WatchSource:0}: Error finding container a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc: Status 404 returned error can't find the container with id a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc Jul 09 19:43:35.817948 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.817924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:35.916204 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.916173 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:35.942731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.942698 2562 generic.go:356] "Generic (PLEG): container finished" podID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerID="f3a372f60130aa488379b1f3b1adf658b502c7dd4e855165d35fde22001594f7" exitCode=0 Jul 09 19:43:35.942850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.942748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" event={"ID":"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5","Type":"ContainerDied","Data":"f3a372f60130aa488379b1f3b1adf658b502c7dd4e855165d35fde22001594f7"} Jul 09 19:43:35.942850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:35.942796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" event={"ID":"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5","Type":"ContainerStarted","Data":"a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc"} Jul 09 19:43:37.925153 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:43:37.925092 2562 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \"/var/tmp/container_images_storage1843609081/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260709%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260709T194337Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=07882c2f1f371cf8afc5c05bd47fc292b28fb362a763c15235b353cd7530ac7f®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1783627117~hmac=9b4b3dd1fc1b1c700691d3e9b3655304c266724787566121f2c7c303b35a3fbb\": EOF); artifact err: provided artifact is a container image" image="registry.redhat.io/leader-worker-set/lws-operator-bundle@sha256:f641cf3b2d6659e26eb0ae49c67a3416152a2fbc2f26f490889a2bd169b9ea06" Jul 09 19:43:37.925555 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:43:37.925224 2562 kuberuntime_manager.go:1459] "Unhandled Error" err="init container pull start failed in pod 77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr_openshift-marketplace(5e3d921d-eff0-492e-ba7e-69e32c2ed2d5): ErrImagePull: unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \"/var/tmp/container_images_storage1843609081/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260709%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260709T194337Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=07882c2f1f371cf8afc5c05bd47fc292b28fb362a763c15235b353cd7530ac7f®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1783627117~hmac=9b4b3dd1fc1b1c700691d3e9b3655304c266724787566121f2c7c303b35a3fbb\": EOF); artifact err: provided artifact is a container image" logger="UnhandledError" Jul 09 19:43:37.925555 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:43:37.925270 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \\\"/var/tmp/container_images_storage1843609081/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260709%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260709T194337Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=07882c2f1f371cf8afc5c05bd47fc292b28fb362a763c15235b353cd7530ac7f®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1783627117~hmac=9b4b3dd1fc1b1c700691d3e9b3655304c266724787566121f2c7c303b35a3fbb\\\": EOF); artifact err: provided artifact is a container image\"" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" Jul 09 19:43:37.952258 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:43:37.952225 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/leader-worker-set/lws-operator-bundle@sha256:f641cf3b2d6659e26eb0ae49c67a3416152a2fbc2f26f490889a2bd169b9ea06\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \\\"/var/tmp/container_images_storage1843609081/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/8e/8ed306a9eb2cdc7b5db57788b7a70b8d6dae2ecfc602b7662ac4cebb197a1841?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260709%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260709T194337Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=07882c2f1f371cf8afc5c05bd47fc292b28fb362a763c15235b353cd7530ac7f®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=leader-worker-set----lws-operator-bundle&akamai_signature=exp=1783627117~hmac=9b4b3dd1fc1b1c700691d3e9b3655304c266724787566121f2c7c303b35a3fbb\\\": EOF); artifact err: provided artifact is a container image\"" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" Jul 09 19:43:38.034701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:38.034667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:38.214113 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:38.214079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:40.269475 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:40.269438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:40.329101 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:40.329070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:46.231669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:46.231635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:47.658193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:47.658157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:43:56.016376 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:56.016342 2562 generic.go:356] "Generic (PLEG): container finished" podID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerID="88dfa77a1d033c0c437e010be58033be9332bd33df827e9193b3d4839f21c369" exitCode=0 Jul 09 19:43:56.016762 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:56.016384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" event={"ID":"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5","Type":"ContainerDied","Data":"88dfa77a1d033c0c437e010be58033be9332bd33df827e9193b3d4839f21c369"} Jul 09 19:43:57.026729 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:57.026695 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" event={"ID":"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5","Type":"ContainerDied","Data":"94ac8929a7441e9cdb32322baeaa640109e7d55127b02473f2975e2095f4a7bb"} Jul 09 19:43:57.026729 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:57.026670 2562 generic.go:356] "Generic (PLEG): container finished" podID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerID="94ac8929a7441e9cdb32322baeaa640109e7d55127b02473f2975e2095f4a7bb" exitCode=0 Jul 09 19:43:58.153115 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.153089 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:58.256602 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.256548 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util\") pod \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " Jul 09 19:43:58.256602 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.256603 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcnjm\" (UniqueName: \"kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm\") pod \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " Jul 09 19:43:58.256847 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.256646 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle\") pod \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\" (UID: \"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5\") " Jul 09 19:43:58.257068 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.257043 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle" (OuterVolumeSpecName: "bundle") pod "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" (UID: "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:43:58.258912 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.258880 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm" (OuterVolumeSpecName: "kube-api-access-hcnjm") pod "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" (UID: "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5"). InnerVolumeSpecName "kube-api-access-hcnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:43:58.261559 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.261534 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util" (OuterVolumeSpecName: "util") pod "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" (UID: "5e3d921d-eff0-492e-ba7e-69e32c2ed2d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:43:58.357563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.357471 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:58.357563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.357503 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnjm\" (UniqueName: \"kubernetes.io/projected/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-kube-api-access-hcnjm\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:58.357563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:58.357517 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e3d921d-eff0-492e-ba7e-69e32c2ed2d5-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:43:59.036749 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:59.036710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" event={"ID":"5e3d921d-eff0-492e-ba7e-69e32c2ed2d5","Type":"ContainerDied","Data":"a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc"} Jul 09 19:43:59.036749 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:59.036744 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjh8dr" Jul 09 19:43:59.036749 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:43:59.036755 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b3e399a9bbbd1809558cc9f7b98b3a8cf3ff13973ef7d4898f0af9b40daedc" Jul 09 19:44:03.008025 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:03.007991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:03.106422 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:03.106389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:03.954142 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:03.954103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:04.372561 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372481 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z"] Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372769 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="pull" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372780 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="pull" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372791 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="util" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372796 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="util" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372804 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="extract" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372809 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="extract" Jul 09 19:44:04.372939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.372871 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3d921d-eff0-492e-ba7e-69e32c2ed2d5" containerName="extract" Jul 09 19:44:04.374940 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.374925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.377631 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.377610 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 19:44:04.377762 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.377734 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:44:04.379049 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.379032 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xkmsl\"" Jul 09 19:44:04.384012 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.383990 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z"] Jul 09 19:44:04.502879 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.502847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece3c198-2336-4fbd-884d-0d33af9b0056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.503057 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.502892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg4m\" (UniqueName: \"kubernetes.io/projected/ece3c198-2336-4fbd-884d-0d33af9b0056-kube-api-access-wzg4m\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.603891 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.603859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece3c198-2336-4fbd-884d-0d33af9b0056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.604085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.603906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg4m\" (UniqueName: \"kubernetes.io/projected/ece3c198-2336-4fbd-884d-0d33af9b0056-kube-api-access-wzg4m\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.604328 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.604307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece3c198-2336-4fbd-884d-0d33af9b0056-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.613312 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.613284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg4m\" (UniqueName: \"kubernetes.io/projected/ece3c198-2336-4fbd-884d-0d33af9b0056-kube-api-access-wzg4m\") pod \"openshift-lws-operator-fd8ccff4c-nxz8z\" (UID: \"ece3c198-2336-4fbd-884d-0d33af9b0056\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.684112 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.684091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" Jul 09 19:44:04.810427 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:04.810392 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z"] Jul 09 19:44:04.812709 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:04.812683 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podece3c198_2336_4fbd_884d_0d33af9b0056.slice/crio-9cb184562b2042af79f81f2a9e2b8a38b87dcc6fdb3e8b69b1c01367565ef04d WatchSource:0}: Error finding container 9cb184562b2042af79f81f2a9e2b8a38b87dcc6fdb3e8b69b1c01367565ef04d: Status 404 returned error can't find the container with id 9cb184562b2042af79f81f2a9e2b8a38b87dcc6fdb3e8b69b1c01367565ef04d Jul 09 19:44:05.065788 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:05.065703 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" event={"ID":"ece3c198-2336-4fbd-884d-0d33af9b0056","Type":"ContainerStarted","Data":"9cb184562b2042af79f81f2a9e2b8a38b87dcc6fdb3e8b69b1c01367565ef04d"} Jul 09 19:44:07.076261 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:07.076221 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" event={"ID":"ece3c198-2336-4fbd-884d-0d33af9b0056","Type":"ContainerStarted","Data":"8f7572469339bdbd8aafb67e898348474e5a7bd21f8d36e2dc712c0d1fe0431d"} Jul 09 19:44:07.094662 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:07.094612 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nxz8z" podStartSLOduration=1.337845338 podStartE2EDuration="3.094594378s" totalImagesPullingTime="1.75674904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:07.092863642 +0000 UTC m=+273.683623847" watchObservedRunningTime="2026-07-09 19:44:07.094594378 +0000 UTC m=+273.685354578" Jul 09 19:44:07.832262 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:07.832231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:07.931200 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:07.931168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:09.430657 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.430628 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg"] Jul 09 19:44:09.434137 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.434122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.436821 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.436804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:44:09.436925 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.436907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jhwl5\"" Jul 09 19:44:09.437060 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.437046 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:44:09.443270 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.443242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg"] Jul 09 19:44:09.444798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.444706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.444798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.444755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9q6\" (UniqueName: \"kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.444957 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.444851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.545987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.545960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.546138 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.545994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9q6\" (UniqueName: \"kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.546138 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.546048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.546318 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.546300 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.546394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.546373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.554291 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.554259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9q6\" (UniqueName: \"kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.594159 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.594119 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:09.745199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.745126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:09.871523 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:09.871492 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg"] Jul 09 19:44:09.874327 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:09.874298 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b17cbdb_15f1_45f0_96d2_9334259a115c.slice/crio-15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421 WatchSource:0}: Error finding container 15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421: Status 404 returned error can't find the container with id 15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421 Jul 09 19:44:10.089211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:10.089103 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerID="6b7e407850e2ea8d81473d665825411e752cf84bd37064dc9571c3463ae53a24" exitCode=0 Jul 09 19:44:10.089211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:10.089165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" event={"ID":"9b17cbdb-15f1-45f0-96d2-9334259a115c","Type":"ContainerDied","Data":"6b7e407850e2ea8d81473d665825411e752cf84bd37064dc9571c3463ae53a24"} Jul 09 19:44:10.089432 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:10.089234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" event={"ID":"9b17cbdb-15f1-45f0-96d2-9334259a115c","Type":"ContainerStarted","Data":"15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421"} Jul 09 19:44:11.094614 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:11.094553 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerID="91a62f5ba04b2f52332a0dc14da3ebd8ecfaaad542ea56df50a6a8c3e5799d23" exitCode=0 Jul 09 19:44:11.094963 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:11.094639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" event={"ID":"9b17cbdb-15f1-45f0-96d2-9334259a115c","Type":"ContainerDied","Data":"91a62f5ba04b2f52332a0dc14da3ebd8ecfaaad542ea56df50a6a8c3e5799d23"} Jul 09 19:44:12.102806 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:12.102776 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerID="911fdd090ecd7b4a753dc809f7d19a07318e6a1248a0f5c6affe7f91fc3f940f" exitCode=0 Jul 09 19:44:12.103145 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:12.102814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" event={"ID":"9b17cbdb-15f1-45f0-96d2-9334259a115c","Type":"ContainerDied","Data":"911fdd090ecd7b4a753dc809f7d19a07318e6a1248a0f5c6affe7f91fc3f940f"} Jul 09 19:44:13.228930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.228907 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:13.274207 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.274178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util\") pod \"9b17cbdb-15f1-45f0-96d2-9334259a115c\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " Jul 09 19:44:13.274408 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.274215 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle\") pod \"9b17cbdb-15f1-45f0-96d2-9334259a115c\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " Jul 09 19:44:13.274408 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.274301 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k9q6\" (UniqueName: \"kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6\") pod \"9b17cbdb-15f1-45f0-96d2-9334259a115c\" (UID: \"9b17cbdb-15f1-45f0-96d2-9334259a115c\") " Jul 09 19:44:13.274935 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.274909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle" (OuterVolumeSpecName: "bundle") pod "9b17cbdb-15f1-45f0-96d2-9334259a115c" (UID: "9b17cbdb-15f1-45f0-96d2-9334259a115c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:44:13.276453 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.276419 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6" (OuterVolumeSpecName: "kube-api-access-4k9q6") pod "9b17cbdb-15f1-45f0-96d2-9334259a115c" (UID: "9b17cbdb-15f1-45f0-96d2-9334259a115c"). InnerVolumeSpecName "kube-api-access-4k9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:44:13.279922 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.279776 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util" (OuterVolumeSpecName: "util") pod "9b17cbdb-15f1-45f0-96d2-9334259a115c" (UID: "9b17cbdb-15f1-45f0-96d2-9334259a115c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:44:13.375096 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.375013 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:13.375096 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.375042 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b17cbdb-15f1-45f0-96d2-9334259a115c-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:13.375096 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:13.375051 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9q6\" (UniqueName: \"kubernetes.io/projected/9b17cbdb-15f1-45f0-96d2-9334259a115c-kube-api-access-4k9q6\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:14.112214 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:14.112183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" event={"ID":"9b17cbdb-15f1-45f0-96d2-9334259a115c","Type":"ContainerDied","Data":"15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421"} Jul 09 19:44:14.112214 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:14.112197 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhnxxg" Jul 09 19:44:14.112454 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:14.112227 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15c19797be320cff84f7d268679d0727c4363a32192874daf11405a807486421" Jul 09 19:44:15.778735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:15.778696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:15.889731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:15.889698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:23.585638 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:23.585603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:23.679385 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:23.679343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.190105 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.190060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.450823 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.450725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.555016 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.554971 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.590598 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590555 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn"] Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590803 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="extract" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590814 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="extract" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590822 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="util" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590828 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="util" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590855 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="pull" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590861 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="pull" Jul 09 19:44:24.590913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.590904 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b17cbdb-15f1-45f0-96d2-9334259a115c" containerName="extract" Jul 09 19:44:24.593617 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.593601 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.597157 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.597132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 19:44:24.597418 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.597404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 19:44:24.597483 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.597462 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 19:44:24.597550 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.597525 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 19:44:24.597885 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.597870 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kw7nt\"" Jul 09 19:44:24.609347 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.609328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn"] Jul 09 19:44:24.653850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.653825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.654009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.653861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-tmp\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.654009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.653888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-manifests\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.654009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.653980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4nfw\" (UniqueName: \"kubernetes.io/projected/174887e9-7aaa-4a90-9538-fc9a29d000d4-kube-api-access-n4nfw\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.654127 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.654022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-webhook-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755116 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-manifests\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755116 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4nfw\" (UniqueName: \"kubernetes.io/projected/174887e9-7aaa-4a90-9538-fc9a29d000d4-kube-api-access-n4nfw\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755116 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-webhook-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755347 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755347 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-tmp\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.755526 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.755499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-manifests\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.757641 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.757613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/174887e9-7aaa-4a90-9538-fc9a29d000d4-tmp\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.757787 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.757768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.757932 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.757912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/174887e9-7aaa-4a90-9538-fc9a29d000d4-webhook-cert\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.773652 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.773620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4nfw\" (UniqueName: \"kubernetes.io/projected/174887e9-7aaa-4a90-9538-fc9a29d000d4-kube-api-access-n4nfw\") pod \"opendatahub-operator-controller-manager-776898495d-vxjqn\" (UID: \"174887e9-7aaa-4a90-9538-fc9a29d000d4\") " pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.775195 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.775168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.862884 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.862849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:24.903738 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.903709 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:24.963635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:24.963517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:25.039877 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.039851 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn"] Jul 09 19:44:25.042317 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:25.042294 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174887e9_7aaa_4a90_9538_fc9a29d000d4.slice/crio-3dab4bf7d8837dedcd6842700ba41b0c3a6ed09375e920709af3ee5700282612 WatchSource:0}: Error finding container 3dab4bf7d8837dedcd6842700ba41b0c3a6ed09375e920709af3ee5700282612: Status 404 returned error can't find the container with id 3dab4bf7d8837dedcd6842700ba41b0c3a6ed09375e920709af3ee5700282612 Jul 09 19:44:25.045758 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.045712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:25.125407 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.125375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:25.156199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.156172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" event={"ID":"174887e9-7aaa-4a90-9538-fc9a29d000d4","Type":"ContainerStarted","Data":"3dab4bf7d8837dedcd6842700ba41b0c3a6ed09375e920709af3ee5700282612"} Jul 09 19:44:25.234633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.234599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:25.326285 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.326197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:25.426176 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:25.426142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:29.174663 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:29.174623 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" event={"ID":"174887e9-7aaa-4a90-9538-fc9a29d000d4","Type":"ContainerStarted","Data":"8da77c35d621054ad8b66e31af25cebd14487723b88d9d8e761d7f128eb96573"} Jul 09 19:44:30.180903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:30.180860 2562 generic.go:356] "Generic (PLEG): container finished" podID="174887e9-7aaa-4a90-9538-fc9a29d000d4" containerID="8da77c35d621054ad8b66e31af25cebd14487723b88d9d8e761d7f128eb96573" exitCode=0 Jul 09 19:44:30.181339 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:30.180951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" event={"ID":"174887e9-7aaa-4a90-9538-fc9a29d000d4","Type":"ContainerDied","Data":"8da77c35d621054ad8b66e31af25cebd14487723b88d9d8e761d7f128eb96573"} Jul 09 19:44:31.187331 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:31.187298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" event={"ID":"174887e9-7aaa-4a90-9538-fc9a29d000d4","Type":"ContainerStarted","Data":"b0c2193dbcdf62ce6dd11faaf7dfdf61d4a0dd869be4a0f5faec7553e1c6da0a"} Jul 09 19:44:31.187830 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:31.187425 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:31.209472 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:31.209426 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" podStartSLOduration=2.637063618 podStartE2EDuration="7.20940925s" totalImagesPullingTime="3.524708722s" totalInitContainerRuntime="1.04763691s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:31.208702088 +0000 UTC m=+297.799462312" watchObservedRunningTime="2026-07-09 19:44:31.20940925 +0000 UTC m=+297.800169455" Jul 09 19:44:33.857553 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:33.857512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:44:33.858391 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:33.858365 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:44:33.867425 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:33.867405 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 19:44:35.775705 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:35.775664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:36.943136 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:36.943097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:42.192904 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:42.192872 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-776898495d-vxjqn" Jul 09 19:44:43.877093 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:43.877059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:44.034332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.034303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:44.350178 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.350140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:44.413064 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.413028 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8"] Jul 09 19:44:44.417390 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.417367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.423680 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.423416 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 19:44:44.424262 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.424244 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 19:44:44.424339 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.424324 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 19:44:44.424380 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.424349 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 19:44:44.461545 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.461512 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8"] Jul 09 19:44:44.511793 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.511764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tmp\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.511793 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.511792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmjc\" (UniqueName: \"kubernetes.io/projected/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-kube-api-access-kqmjc\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.511965 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.511885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tls-certs\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.612287 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.612212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tls-certs\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.612287 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.612254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tmp\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.612287 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.612278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmjc\" (UniqueName: \"kubernetes.io/projected/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-kube-api-access-kqmjc\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.614865 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.614834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tls-certs\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.614976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.614868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-tmp\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.639262 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.639233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmjc\" (UniqueName: \"kubernetes.io/projected/24f55663-8dfb-43b4-a2e4-06fb45a5a38b-kube-api-access-kqmjc\") pod \"kube-auth-proxy-d5f4bc74d-x2hz8\" (UID: \"24f55663-8dfb-43b4-a2e4-06fb45a5a38b\") " pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.726255 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.726225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" Jul 09 19:44:44.756209 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.756174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:44.764246 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.764224 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ncn95"] Jul 09 19:44:44.768286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.768263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:44.770955 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.770936 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 19:44:44.771295 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.771278 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qjsxt\"" Jul 09 19:44:44.783651 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.783631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ncn95"] Jul 09 19:44:44.852870 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.852830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:44.877635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.877615 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8"] Jul 09 19:44:44.879528 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:44.879507 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f55663_8dfb_43b4_a2e4_06fb45a5a38b.slice/crio-3b58335f56bfdcad69f72f2846f59867545e9e932c8cee6015abf23b89b758b9 WatchSource:0}: Error finding container 3b58335f56bfdcad69f72f2846f59867545e9e932c8cee6015abf23b89b758b9: Status 404 returned error can't find the container with id 3b58335f56bfdcad69f72f2846f59867545e9e932c8cee6015abf23b89b758b9 Jul 09 19:44:44.881203 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.881187 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:44:44.914862 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.914839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8r7\" (UniqueName: \"kubernetes.io/projected/7cedd294-177f-46b8-996c-5150e85d805b-kube-api-access-9h8r7\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:44.914984 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.914879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:44.944575 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:44.944545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:45.016123 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.016085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.016281 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.016186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8r7\" (UniqueName: \"kubernetes.io/projected/7cedd294-177f-46b8-996c-5150e85d805b-kube-api-access-9h8r7\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.016281 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:44:45.016253 2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 19:44:45.016352 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:44:45.016313 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert podName:7cedd294-177f-46b8-996c-5150e85d805b nodeName:}" failed. No retries permitted until 2026-07-09 19:44:45.516298893 +0000 UTC m=+312.107059076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert") pod "odh-model-controller-7998c67dc4-ncn95" (UID: "7cedd294-177f-46b8-996c-5150e85d805b") : secret "odh-model-controller-webhook-cert" not found Jul 09 19:44:45.029784 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.029753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8r7\" (UniqueName: \"kubernetes.io/projected/7cedd294-177f-46b8-996c-5150e85d805b-kube-api-access-9h8r7\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.032070 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.032047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:45.223591 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.223555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:45.250725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.250690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" event={"ID":"24f55663-8dfb-43b4-a2e4-06fb45a5a38b","Type":"ContainerStarted","Data":"3b58335f56bfdcad69f72f2846f59867545e9e932c8cee6015abf23b89b758b9"} Jul 09 19:44:45.521335 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.521237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.523872 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.523846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cedd294-177f-46b8-996c-5150e85d805b-cert\") pod \"odh-model-controller-7998c67dc4-ncn95\" (UID: \"7cedd294-177f-46b8-996c-5150e85d805b\") " pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.682677 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.682644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:45.828188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.828095 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ncn95"] Jul 09 19:44:45.831200 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:45.831155 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cedd294_177f_46b8_996c_5150e85d805b.slice/crio-04d66317fee7d9ce2b72d7609ef5942639ce8f359366611001f658eb012c8ed7 WatchSource:0}: Error finding container 04d66317fee7d9ce2b72d7609ef5942639ce8f359366611001f658eb012c8ed7: Status 404 returned error can't find the container with id 04d66317fee7d9ce2b72d7609ef5942639ce8f359366611001f658eb012c8ed7 Jul 09 19:44:45.871008 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.870975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:45.883054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.883027 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g5fl4"] Jul 09 19:44:45.887903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.887882 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:45.897301 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.897282 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 09 19:44:45.897409 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.897318 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-thnkq\"" Jul 09 19:44:45.903265 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.903244 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g5fl4"] Jul 09 19:44:45.992206 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.992095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:45.997653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:45.997629 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:46.001080 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.001058 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.005885 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.005866 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 19:44:46.005885 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.005910 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 19:44:46.006187 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.006168 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 19:44:46.006673 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.006652 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 19:44:46.006771 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.006686 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2cnct\"" Jul 09 19:44:46.024656 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.024635 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:46.025789 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.025770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-tls-certs\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.025872 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.025805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9v9x\" (UniqueName: \"kubernetes.io/projected/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-kube-api-access-w9v9x\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.087597 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.087498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:46.126461 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-tls-certs\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.126601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126737 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgp4t\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126737 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9v9x\" (UniqueName: \"kubernetes.io/projected/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-kube-api-access-w9v9x\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.126737 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126834 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.126834 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.126791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.128956 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.128939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-tls-certs\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.135478 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.135460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9v9x\" (UniqueName: \"kubernetes.io/projected/5faa2b79-08e6-46e1-a4d8-94ab0068a98b-kube-api-access-w9v9x\") pod \"model-serving-api-54d89bd5fc-g5fl4\" (UID: \"5faa2b79-08e6-46e1-a4d8-94ab0068a98b\") " pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.199467 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.199441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:46.227830 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.227968 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.227968 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.227968 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.227968 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.228164 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.227981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.228164 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.228012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgp4t\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.229039 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.229013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.230289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.230240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:46.231362 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.231335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.231362 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.231352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.231555 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.231537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.232408 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.232377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.239669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.239643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.240092 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.240072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgp4t\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t\") pod \"istiod-openshift-gateway-7574459656-9ghsm\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.257864 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.257794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" event={"ID":"7cedd294-177f-46b8-996c-5150e85d805b","Type":"ContainerStarted","Data":"04d66317fee7d9ce2b72d7609ef5942639ce8f359366611001f658eb012c8ed7"} Jul 09 19:44:46.312015 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.311986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:46.339498 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.339475 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g5fl4"] Jul 09 19:44:46.341415 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:46.341389 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faa2b79_08e6_46e1_a4d8_94ab0068a98b.slice/crio-99bed6965c28eb0fc7046517408e29027bcff08423a1603beba112f60a2b4822 WatchSource:0}: Error finding container 99bed6965c28eb0fc7046517408e29027bcff08423a1603beba112f60a2b4822: Status 404 returned error can't find the container with id 99bed6965c28eb0fc7046517408e29027bcff08423a1603beba112f60a2b4822 Jul 09 19:44:46.464447 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:46.464423 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:46.466787 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:46.466757 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda595c0f3_eaf0_4666_911f_aa104c2c52d2.slice/crio-ba3b9f355f72a6e58a9ac5d752db8ed64bdee760d52e5c7736acbb0acb602618 WatchSource:0}: Error finding container ba3b9f355f72a6e58a9ac5d752db8ed64bdee760d52e5c7736acbb0acb602618: Status 404 returned error can't find the container with id ba3b9f355f72a6e58a9ac5d752db8ed64bdee760d52e5c7736acbb0acb602618 Jul 09 19:44:47.268171 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:47.268130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" event={"ID":"5faa2b79-08e6-46e1-a4d8-94ab0068a98b","Type":"ContainerStarted","Data":"99bed6965c28eb0fc7046517408e29027bcff08423a1603beba112f60a2b4822"} Jul 09 19:44:47.270206 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:47.270131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" event={"ID":"a595c0f3-eaf0-4666-911f-aa104c2c52d2","Type":"ContainerStarted","Data":"ba3b9f355f72a6e58a9ac5d752db8ed64bdee760d52e5c7736acbb0acb602618"} Jul 09 19:44:47.381855 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:47.381790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:50.899332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:50.899297 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:44:50.899701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:50.899360 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:44:51.293094 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.293055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" event={"ID":"24f55663-8dfb-43b4-a2e4-06fb45a5a38b","Type":"ContainerStarted","Data":"ad3fce696835f86a99312e7dbd06d8e31b3c6ef5e2c4001c08c5fca589445cb0"} Jul 09 19:44:51.294830 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.294791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" event={"ID":"7cedd294-177f-46b8-996c-5150e85d805b","Type":"ContainerStarted","Data":"1cc135ee6e883606a4e7ad570e5c3d9d45d7fe97793832aac5d74f74a6e93203"} Jul 09 19:44:51.295028 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.294853 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:44:51.296327 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.296303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" event={"ID":"5faa2b79-08e6-46e1-a4d8-94ab0068a98b","Type":"ContainerStarted","Data":"ef3278f794c17fb66e56acef10061e8b18918f471e9367e378f679df836a0937"} Jul 09 19:44:51.296471 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.296348 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:44:51.298020 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.297999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" event={"ID":"a595c0f3-eaf0-4666-911f-aa104c2c52d2","Type":"ContainerStarted","Data":"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb"} Jul 09 19:44:51.298182 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.298170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:51.299907 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.299883 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9ghsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 19:44:51.299991 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.299934 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:44:51.312342 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.312296 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d5f4bc74d-x2hz8" podStartSLOduration=1.30130079 podStartE2EDuration="7.312283785s" totalImagesPullingTime="6.010982995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:51.310711453 +0000 UTC m=+317.901471657" watchObservedRunningTime="2026-07-09 19:44:51.312283785 +0000 UTC m=+317.903043990" Jul 09 19:44:51.330617 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.330551 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" podStartSLOduration=1.77724733 podStartE2EDuration="6.330533052s" totalImagesPullingTime="4.553285722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:51.32893703 +0000 UTC m=+317.919697241" watchObservedRunningTime="2026-07-09 19:44:51.330533052 +0000 UTC m=+317.921293259" Jul 09 19:44:51.358147 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.358088 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" podStartSLOduration=1.928428948 podStartE2EDuration="6.358067328s" totalImagesPullingTime="4.42963838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:51.356112771 +0000 UTC m=+317.946872997" watchObservedRunningTime="2026-07-09 19:44:51.358067328 +0000 UTC m=+317.948827534" Jul 09 19:44:51.374839 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.374787 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" podStartSLOduration=2.298845775 podStartE2EDuration="7.374767151s" totalImagesPullingTime="5.075921376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:51.374062456 +0000 UTC m=+317.964822657" watchObservedRunningTime="2026-07-09 19:44:51.374767151 +0000 UTC m=+317.965527358" Jul 09 19:44:51.865063 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.865020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:51.984790 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:51.984748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:52.303743 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.303713 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9ghsm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 19:44:52.303930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.303767 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:44:52.721510 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.721479 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh"] Jul 09 19:44:52.724796 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.724780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.727850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.727827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pxnlg\"" Jul 09 19:44:52.741830 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.741808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh"] Jul 09 19:44:52.769177 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.769144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:52.887922 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.887890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.887922 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.887926 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.887964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888154 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgmx\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-kube-api-access-nsgmx\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888309 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.888309 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.888181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.989533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.989533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgmx\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-kube-api-access-nsgmx\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.989834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.990092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990546 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.990197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990546 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.990197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990546 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.990391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.990770 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.990747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.992647 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.992623 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:52.992752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:52.992729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:53.000457 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.000434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:53.000540 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.000464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgmx\" (UniqueName: \"kubernetes.io/projected/e0a9fec9-fa38-4c3b-91d5-7d03a6026c97-kube-api-access-nsgmx\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678vnbbh\" (UID: \"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:53.034626 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.034591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:53.059104 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.059064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:53.185596 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.185553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh"] Jul 09 19:44:53.187535 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:53.187501 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a9fec9_fa38_4c3b_91d5_7d03a6026c97.slice/crio-8f3a47ac844cf8968bbcc677bbff00611d1a92fb937131f410a02a149150c868 WatchSource:0}: Error finding container 8f3a47ac844cf8968bbcc677bbff00611d1a92fb937131f410a02a149150c868: Status 404 returned error can't find the container with id 8f3a47ac844cf8968bbcc677bbff00611d1a92fb937131f410a02a149150c868 Jul 09 19:44:53.227054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.227013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:53.307944 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:53.307861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" event={"ID":"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97","Type":"ContainerStarted","Data":"8f3a47ac844cf8968bbcc677bbff00611d1a92fb937131f410a02a149150c868"} Jul 09 19:44:55.304078 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.304047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:55.379334 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.379252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:55.521627 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.521593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:55.524170 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.524147 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q"] Jul 09 19:44:55.527282 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.527265 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.530373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.530352 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn"] Jul 09 19:44:55.531675 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.531655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rz7n7\"" Jul 09 19:44:55.532216 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.532201 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 19:44:55.533772 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.533757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.536836 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.536818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 19:44:55.536927 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.536837 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-whmtr\"" Jul 09 19:44:55.550231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.550213 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn"] Jul 09 19:44:55.551876 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.551856 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q"] Jul 09 19:44:55.577613 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.577569 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:44:55.577701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.577649 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:44:55.577701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.577678 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:44:55.617502 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.617470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkl4h\" (UniqueName: \"kubernetes.io/projected/e3db049e-97d4-4c00-a24f-ba5272bfc88f-kube-api-access-hkl4h\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.617681 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.617547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.617681 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.617650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.617768 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.617709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-kube-api-access-ttcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.640487 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.640456 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:55.718942 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.718909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.719090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.718961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.719090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.718995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-kube-api-access-ttcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.719090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.719030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkl4h\" (UniqueName: \"kubernetes.io/projected/e3db049e-97d4-4c00-a24f-ba5272bfc88f-kube-api-access-hkl4h\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.719253 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:44:55.719095 2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 09 19:44:55.719253 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:44:55.719180 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert podName:e3db049e-97d4-4c00-a24f-ba5272bfc88f nodeName:}" failed. No retries permitted until 2026-07-09 19:44:56.219158912 +0000 UTC m=+322.809919100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert") pod "kserve-controller-manager-d6f4b65b8-4pn8q" (UID: "e3db049e-97d4-4c00-a24f-ba5272bfc88f") : secret "kserve-webhook-server-cert" not found Jul 09 19:44:55.721689 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.721661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.739376 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.739347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/6f65dcb6-a83e-4aed-a5f0-abd60438e4f2-kube-api-access-ttcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-khxfn\" (UID: \"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:55.741468 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.741381 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkl4h\" (UniqueName: \"kubernetes.io/projected/e3db049e-97d4-4c00-a24f-ba5272bfc88f-kube-api-access-hkl4h\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:55.744365 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.744324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:55.845788 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.845689 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:55.846468 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:55.846451 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:44:56.007882 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.007855 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn"] Jul 09 19:44:56.010438 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:56.010410 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f65dcb6_a83e_4aed_a5f0_abd60438e4f2.slice/crio-d72224d9447b390c2015a0d5e4b2dbcd797db83f5e85d5596f10c6549b53e5c0 WatchSource:0}: Error finding container d72224d9447b390c2015a0d5e4b2dbcd797db83f5e85d5596f10c6549b53e5c0: Status 404 returned error can't find the container with id d72224d9447b390c2015a0d5e4b2dbcd797db83f5e85d5596f10c6549b53e5c0 Jul 09 19:44:56.222684 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.222653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:56.225290 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.225261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3db049e-97d4-4c00-a24f-ba5272bfc88f-cert\") pod \"kserve-controller-manager-d6f4b65b8-4pn8q\" (UID: \"e3db049e-97d4-4c00-a24f-ba5272bfc88f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:56.325174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.325143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" event={"ID":"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2","Type":"ContainerStarted","Data":"d72224d9447b390c2015a0d5e4b2dbcd797db83f5e85d5596f10c6549b53e5c0"} Jul 09 19:44:56.326568 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.326539 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" event={"ID":"e0a9fec9-fa38-4c3b-91d5-7d03a6026c97","Type":"ContainerStarted","Data":"37e216b3a0c83aefe57b6681a88e964f0740a9c05bf178e794bc22480fdfadcd"} Jul 09 19:44:56.351937 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.351879 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" podStartSLOduration=1.9643940180000001 podStartE2EDuration="4.351864014s" totalImagesPullingTime="2.387469996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:44:56.349028471 +0000 UTC m=+322.939788715" watchObservedRunningTime="2026-07-09 19:44:56.351864014 +0000 UTC m=+322.942624218" Jul 09 19:44:56.441196 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.441164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:44:56.597530 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.597507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q"] Jul 09 19:44:56.598862 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:44:56.598837 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3db049e_97d4_4c00_a24f_ba5272bfc88f.slice/crio-426e213b3e6a995a6e9ac04e4e9504d5f7695a561ab315a5524785023277137d WatchSource:0}: Error finding container 426e213b3e6a995a6e9ac04e4e9504d5f7695a561ab315a5524785023277137d: Status 404 returned error can't find the container with id 426e213b3e6a995a6e9ac04e4e9504d5f7695a561ab315a5524785023277137d Jul 09 19:44:56.783978 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.783900 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:56.784184 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.784154 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" containerID="cri-o://790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb" gracePeriod=30 Jul 09 19:44:56.814540 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.814510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:56.904298 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:56.904263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:57.004832 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.004790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:44:57.035739 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.035668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:57.040065 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.040043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:57.311961 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.311940 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:57.333664 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.333633 2562 generic.go:356] "Generic (PLEG): container finished" podID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerID="790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb" exitCode=0 Jul 09 19:44:57.334058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.333674 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" Jul 09 19:44:57.334058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.333712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" event={"ID":"a595c0f3-eaf0-4666-911f-aa104c2c52d2","Type":"ContainerDied","Data":"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb"} Jul 09 19:44:57.334058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.333776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm" event={"ID":"a595c0f3-eaf0-4666-911f-aa104c2c52d2","Type":"ContainerDied","Data":"ba3b9f355f72a6e58a9ac5d752db8ed64bdee760d52e5c7736acbb0acb602618"} Jul 09 19:44:57.334058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.333808 2562 scope.go:117] "RemoveContainer" containerID="790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb" Jul 09 19:44:57.335279 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.335201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" event={"ID":"e3db049e-97d4-4c00-a24f-ba5272bfc88f","Type":"ContainerStarted","Data":"426e213b3e6a995a6e9ac04e4e9504d5f7695a561ab315a5524785023277137d"} Jul 09 19:44:57.336731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.336411 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:57.337659 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.337641 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678vnbbh" Jul 09 19:44:57.348534 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.348519 2562 scope.go:117] "RemoveContainer" containerID="790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb" Jul 09 19:44:57.348878 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:44:57.348846 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb\": container with ID starting with 790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb not found: ID does not exist" containerID="790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb" Jul 09 19:44:57.348951 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.348877 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb"} err="failed to get container status \"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb\": rpc error: code = NotFound desc = could not find container \"790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb\": container with ID starting with 790495c515a9f4ab7bf085855af6f88d29e16360375367bc8315cf0eb9400ccb not found: ID does not exist" Jul 09 19:44:57.443630 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443575 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.443783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgp4t\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.443783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443683 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.443783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.443783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443735 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.443783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443779 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.444056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.443817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap\") pod \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\" (UID: \"a595c0f3-eaf0-4666-911f-aa104c2c52d2\") " Jul 09 19:44:57.444711 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.444393 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:44:57.446798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.446729 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t" (OuterVolumeSpecName: "kube-api-access-sgp4t") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "kube-api-access-sgp4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:44:57.446798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.446745 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs" (OuterVolumeSpecName: "local-certs") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:44:57.446798 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.446731 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:44:57.447160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.447126 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:44:57.447286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.447163 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts" (OuterVolumeSpecName: "cacerts") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:44:57.447553 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.447528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a595c0f3-eaf0-4666-911f-aa104c2c52d2" (UID: "a595c0f3-eaf0-4666-911f-aa104c2c52d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544828 2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-cacerts\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544855 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-ca-configmap\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544866 2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a595c0f3-eaf0-4666-911f-aa104c2c52d2-local-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544875 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgp4t\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-kube-api-access-sgp4t\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544883 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-csr-dns-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544891 2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-kubeconfig\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.544905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.544900 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a595c0f3-eaf0-4666-911f-aa104c2c52d2-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:44:57.660657 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.660625 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:57.664818 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.664787 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9ghsm"] Jul 09 19:44:57.954007 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:44:57.953971 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" path="/var/lib/kubelet/pods/a595c0f3-eaf0-4666-911f-aa104c2c52d2/volumes" Jul 09 19:45:00.353498 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.353460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" event={"ID":"e3db049e-97d4-4c00-a24f-ba5272bfc88f","Type":"ContainerStarted","Data":"3990bdd372f845445812e093032c3624139ce2baac2d027a6886a351c1bc3537"} Jul 09 19:45:00.353958 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.353528 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:45:00.355160 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.355111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" event={"ID":"6f65dcb6-a83e-4aed-a5f0-abd60438e4f2","Type":"ContainerStarted","Data":"38dc0276631a0f06787353ec2e7640d22810cca61fc3fc7208907bc7a7e1ed47"} Jul 09 19:45:00.355273 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.355201 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:45:00.370857 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.370812 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" podStartSLOduration=2.153791195 podStartE2EDuration="5.370795967s" totalImagesPullingTime="3.217004772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:45:00.370477618 +0000 UTC m=+326.961237853" watchObservedRunningTime="2026-07-09 19:45:00.370795967 +0000 UTC m=+326.961556173" Jul 09 19:45:00.389193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:00.389152 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" podStartSLOduration=1.585730549 podStartE2EDuration="5.389141105s" totalImagesPullingTime="3.803410556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:45:00.388476606 +0000 UTC m=+326.979236835" watchObservedRunningTime="2026-07-09 19:45:00.389141105 +0000 UTC m=+326.979901295" Jul 09 19:45:01.320043 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:01.320004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:01.420606 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:01.420548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:02.305359 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:02.305331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ncn95" Jul 09 19:45:02.307199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:02.307179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-g5fl4" Jul 09 19:45:21.455888 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:21.455853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:21.514406 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:21.514366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:31.362075 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:31.361985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-khxfn" Jul 09 19:45:31.365068 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:31.365052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4pn8q" Jul 09 19:45:45.324738 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.324683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:45.412666 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.412624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:45.453730 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.453703 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:45.454016 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.454005 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" Jul 09 19:45:45.454063 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.454018 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" Jul 09 19:45:45.454096 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.454071 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a595c0f3-eaf0-4666-911f-aa104c2c52d2" containerName="discovery" Jul 09 19:45:45.461838 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.461809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:45.465212 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.465185 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 19:45:45.465681 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.465657 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 19:45:45.466702 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.466678 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-jvp24\"" Jul 09 19:45:45.469631 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.469542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:45.511638 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.511600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:45.537475 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.537451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4q9\" (UniqueName: \"kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9\") pod \"kuadrant-operator-catalog-x2dnv\" (UID: \"3ea594c8-07e4-40b0-ae8a-e01ee76b4508\") " pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:45.638719 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.638692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4q9\" (UniqueName: \"kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9\") pod \"kuadrant-operator-catalog-x2dnv\" (UID: \"3ea594c8-07e4-40b0-ae8a-e01ee76b4508\") " pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:45.647595 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.647563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4q9\" (UniqueName: \"kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9\") pod \"kuadrant-operator-catalog-x2dnv\" (UID: \"3ea594c8-07e4-40b0-ae8a-e01ee76b4508\") " pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:45.773889 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.773855 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:45.812031 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.811701 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:45.906807 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:45.906690 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:45.909102 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:45:45.909077 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea594c8_07e4_40b0_ae8a_e01ee76b4508.slice/crio-a09b3b8cdd138668e3c8473dd7a26a045c8f0e1322b0dce3a55c70e8d235fc45 WatchSource:0}: Error finding container a09b3b8cdd138668e3c8473dd7a26a045c8f0e1322b0dce3a55c70e8d235fc45: Status 404 returned error can't find the container with id a09b3b8cdd138668e3c8473dd7a26a045c8f0e1322b0dce3a55c70e8d235fc45 Jul 09 19:45:46.023849 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.023823 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5p2h8"] Jul 09 19:45:46.026790 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.026772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:46.038678 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.038659 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5p2h8"] Jul 09 19:45:46.142227 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.142198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmc4\" (UniqueName: \"kubernetes.io/projected/c7c9a28e-de41-43e3-bce5-2dcbb62b32c9-kube-api-access-hbmc4\") pod \"kuadrant-operator-catalog-5p2h8\" (UID: \"c7c9a28e-de41-43e3-bce5-2dcbb62b32c9\") " pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:46.243708 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.243619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmc4\" (UniqueName: \"kubernetes.io/projected/c7c9a28e-de41-43e3-bce5-2dcbb62b32c9-kube-api-access-hbmc4\") pod \"kuadrant-operator-catalog-5p2h8\" (UID: \"c7c9a28e-de41-43e3-bce5-2dcbb62b32c9\") " pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:46.253250 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.253225 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmc4\" (UniqueName: \"kubernetes.io/projected/c7c9a28e-de41-43e3-bce5-2dcbb62b32c9-kube-api-access-hbmc4\") pod \"kuadrant-operator-catalog-5p2h8\" (UID: \"c7c9a28e-de41-43e3-bce5-2dcbb62b32c9\") " pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:46.266220 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.266192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:46.336415 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.336385 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:46.456374 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.456338 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:45:46.481500 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.481400 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5p2h8"] Jul 09 19:45:46.484007 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:45:46.483980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c9a28e_de41_43e3_bce5_2dcbb62b32c9.slice/crio-55f7b907e090cf46a8aea886d54aaa13c4f4b6aa81490587d6f029043f6a818c WatchSource:0}: Error finding container 55f7b907e090cf46a8aea886d54aaa13c4f4b6aa81490587d6f029043f6a818c: Status 404 returned error can't find the container with id 55f7b907e090cf46a8aea886d54aaa13c4f4b6aa81490587d6f029043f6a818c Jul 09 19:45:46.570548 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.570458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" event={"ID":"c7c9a28e-de41-43e3-bce5-2dcbb62b32c9","Type":"ContainerStarted","Data":"55f7b907e090cf46a8aea886d54aaa13c4f4b6aa81490587d6f029043f6a818c"} Jul 09 19:45:46.571826 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:46.571800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" event={"ID":"3ea594c8-07e4-40b0-ae8a-e01ee76b4508","Type":"ContainerStarted","Data":"a09b3b8cdd138668e3c8473dd7a26a045c8f0e1322b0dce3a55c70e8d235fc45"} Jul 09 19:45:48.584018 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.583976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" event={"ID":"c7c9a28e-de41-43e3-bce5-2dcbb62b32c9","Type":"ContainerStarted","Data":"30314a41fe8ed5c202825266b59ecf94e93a86fe3a073ff229998c6a707aee77"} Jul 09 19:45:48.585462 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.585441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" event={"ID":"3ea594c8-07e4-40b0-ae8a-e01ee76b4508","Type":"ContainerStarted","Data":"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd"} Jul 09 19:45:48.585562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.585524 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" podUID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" containerName="registry-server" containerID="cri-o://698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd" gracePeriod=2 Jul 09 19:45:48.604229 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.604184 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" podStartSLOduration=1.121750425 podStartE2EDuration="2.604170448s" totalImagesPullingTime="1.482420023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:45:48.602425016 +0000 UTC m=+375.193185220" watchObservedRunningTime="2026-07-09 19:45:48.604170448 +0000 UTC m=+375.194930653" Jul 09 19:45:48.619874 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.619832 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" podStartSLOduration=1.5655607809999998 podStartE2EDuration="3.619818088s" totalImagesPullingTime="2.054257307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:45:48.619756262 +0000 UTC m=+375.210516493" watchObservedRunningTime="2026-07-09 19:45:48.619818088 +0000 UTC m=+375.210578293" Jul 09 19:45:48.818896 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.818867 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:48.968342 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.968313 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4q9\" (UniqueName: \"kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9\") pod \"3ea594c8-07e4-40b0-ae8a-e01ee76b4508\" (UID: \"3ea594c8-07e4-40b0-ae8a-e01ee76b4508\") " Jul 09 19:45:48.970546 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:48.970516 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9" (OuterVolumeSpecName: "kube-api-access-fz4q9") pod "3ea594c8-07e4-40b0-ae8a-e01ee76b4508" (UID: "3ea594c8-07e4-40b0-ae8a-e01ee76b4508"). InnerVolumeSpecName "kube-api-access-fz4q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:45:49.069150 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.069109 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4q9\" (UniqueName: \"kubernetes.io/projected/3ea594c8-07e4-40b0-ae8a-e01ee76b4508-kube-api-access-fz4q9\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:45:49.591220 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.591181 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" containerID="698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd" exitCode=0 Jul 09 19:45:49.591220 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.591222 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" Jul 09 19:45:49.591786 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.591267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" event={"ID":"3ea594c8-07e4-40b0-ae8a-e01ee76b4508","Type":"ContainerDied","Data":"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd"} Jul 09 19:45:49.591786 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.591316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x2dnv" event={"ID":"3ea594c8-07e4-40b0-ae8a-e01ee76b4508","Type":"ContainerDied","Data":"a09b3b8cdd138668e3c8473dd7a26a045c8f0e1322b0dce3a55c70e8d235fc45"} Jul 09 19:45:49.591786 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.591339 2562 scope.go:117] "RemoveContainer" containerID="698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd" Jul 09 19:45:49.600823 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.600807 2562 scope.go:117] "RemoveContainer" containerID="698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd" Jul 09 19:45:49.601077 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:45:49.601057 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd\": container with ID starting with 698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd not found: ID does not exist" containerID="698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd" Jul 09 19:45:49.601177 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.601083 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd"} err="failed to get container status \"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd\": rpc error: code = NotFound desc = could not find container \"698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd\": container with ID starting with 698ef77121dcc2301609cc67bc0035a5b55db6f7776697af96ec7a46956cd4cd not found: ID does not exist" Jul 09 19:45:49.616360 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.616336 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:49.623549 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.623524 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x2dnv"] Jul 09 19:45:49.952509 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:49.952482 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" path="/var/lib/kubelet/pods/3ea594c8-07e4-40b0-ae8a-e01ee76b4508/volumes" Jul 09 19:45:56.336627 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:56.336575 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:56.337044 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:56.336654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:56.358976 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:56.358947 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:45:56.646495 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:45:56.646472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-5p2h8" Jul 09 19:46:01.064378 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.064338 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b"] Jul 09 19:46:01.064771 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.064707 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" containerName="registry-server" Jul 09 19:46:01.064771 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.064719 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" containerName="registry-server" Jul 09 19:46:01.064844 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.064776 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea594c8-07e4-40b0-ae8a-e01ee76b4508" containerName="registry-server" Jul 09 19:46:01.067020 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.067004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.069808 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.069789 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62ctk\"" Jul 09 19:46:01.079353 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.079325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b"] Jul 09 19:46:01.165552 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.165522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2h2b\" (UniqueName: \"kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.165714 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.165556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.165714 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.165575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.266315 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.266287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2h2b\" (UniqueName: \"kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.266467 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.266328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.266467 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.266358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.266729 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.266711 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.266787 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.266766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.275903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.275882 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2h2b\" (UniqueName: \"kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.375732 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.375649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:01.473396 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.473366 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6"] Jul 09 19:46:01.477325 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.477302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.486366 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.486303 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6"] Jul 09 19:46:01.514460 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.514404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b"] Jul 09 19:46:01.516841 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:01.516812 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf349fa74_6d5b_4d91_aeec_6acf384487c6.slice/crio-02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919 WatchSource:0}: Error finding container 02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919: Status 404 returned error can't find the container with id 02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919 Jul 09 19:46:01.569369 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.569342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxt4n\" (UniqueName: \"kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.569500 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.569385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.569500 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.569406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.648127 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.648084 2562 generic.go:356] "Generic (PLEG): container finished" podID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerID="ab6f7ac0d794f35f3ec61f43dd372b9aaab4f57165b97d10d202b9b6335c18d3" exitCode=0 Jul 09 19:46:01.648292 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.648152 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" event={"ID":"f349fa74-6d5b-4d91-aeec-6acf384487c6","Type":"ContainerDied","Data":"ab6f7ac0d794f35f3ec61f43dd372b9aaab4f57165b97d10d202b9b6335c18d3"} Jul 09 19:46:01.648292 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.648210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" event={"ID":"f349fa74-6d5b-4d91-aeec-6acf384487c6","Type":"ContainerStarted","Data":"02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919"} Jul 09 19:46:01.669984 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.669954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxt4n\" (UniqueName: \"kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.670127 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.669994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.670127 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.670014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.670326 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.670311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.670378 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.670360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.680068 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.680045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxt4n\" (UniqueName: \"kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.789556 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.789521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:01.921857 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.921787 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm"] Jul 09 19:46:01.924989 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.924972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:01.930456 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.930434 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6"] Jul 09 19:46:01.933044 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:01.933015 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7697830a_cdf4_42bb_839c_699a74718ce5.slice/crio-a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3 WatchSource:0}: Error finding container a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3: Status 404 returned error can't find the container with id a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3 Jul 09 19:46:01.938678 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:01.938656 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm"] Jul 09 19:46:02.073340 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.073310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.073752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.073352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7vl\" (UniqueName: \"kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.073752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.073468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.174207 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.174119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.174207 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.174181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.174407 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.174209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7vl\" (UniqueName: \"kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.174622 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.174572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.174664 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.174572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.187072 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.187042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7vl\" (UniqueName: \"kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.235861 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.235827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:02.294924 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.294893 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m"] Jul 09 19:46:02.297732 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.297716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.310221 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.310196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m"] Jul 09 19:46:02.476831 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.476801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.476996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.476879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.476996 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.476900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lk2k\" (UniqueName: \"kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.499814 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.499791 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm"] Jul 09 19:46:02.535383 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:02.535360 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5a4166_2d55_414f_a876_28017499e692.slice/crio-2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216 WatchSource:0}: Error finding container 2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216: Status 404 returned error can't find the container with id 2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216 Jul 09 19:46:02.577754 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.577734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.577846 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.577792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.577846 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.577820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lk2k\" (UniqueName: \"kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.578082 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.578064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.578210 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.578112 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.589790 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.589767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lk2k\" (UniqueName: \"kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.607653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.607628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:02.655204 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.655170 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a5a4166-2d55-414f-a876-28017499e692" containerID="6cc2287f4fbfc20ca4ae8df3ee3db971da7cda58a3d8b8eaeb97519caa6855f7" exitCode=0 Jul 09 19:46:02.655360 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.655258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" event={"ID":"0a5a4166-2d55-414f-a876-28017499e692","Type":"ContainerDied","Data":"6cc2287f4fbfc20ca4ae8df3ee3db971da7cda58a3d8b8eaeb97519caa6855f7"} Jul 09 19:46:02.655441 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.655355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" event={"ID":"0a5a4166-2d55-414f-a876-28017499e692","Type":"ContainerStarted","Data":"2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216"} Jul 09 19:46:02.657184 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.657157 2562 generic.go:356] "Generic (PLEG): container finished" podID="7697830a-cdf4-42bb-839c-699a74718ce5" containerID="9c0d1e2783e3d384cc49a68d33706a0c83617de0e185002b3ada5f60320303e0" exitCode=0 Jul 09 19:46:02.657295 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.657192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" event={"ID":"7697830a-cdf4-42bb-839c-699a74718ce5","Type":"ContainerDied","Data":"9c0d1e2783e3d384cc49a68d33706a0c83617de0e185002b3ada5f60320303e0"} Jul 09 19:46:02.657295 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.657238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" event={"ID":"7697830a-cdf4-42bb-839c-699a74718ce5","Type":"ContainerStarted","Data":"a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3"} Jul 09 19:46:02.659466 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.659439 2562 generic.go:356] "Generic (PLEG): container finished" podID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerID="e8d3c983e9cb6bc9cb22d8da3bd3817d16f3f09d2464f3d0ab4efdcb77a97c52" exitCode=0 Jul 09 19:46:02.659557 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.659472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" event={"ID":"f349fa74-6d5b-4d91-aeec-6acf384487c6","Type":"ContainerDied","Data":"e8d3c983e9cb6bc9cb22d8da3bd3817d16f3f09d2464f3d0ab4efdcb77a97c52"} Jul 09 19:46:02.758821 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:02.758721 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m"] Jul 09 19:46:02.760859 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:02.760826 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a4ed1a_9efd_4976_8732_799ab79c5e6b.slice/crio-c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192 WatchSource:0}: Error finding container c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192: Status 404 returned error can't find the container with id c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192 Jul 09 19:46:03.666677 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.666641 2562 generic.go:356] "Generic (PLEG): container finished" podID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerID="53d0fe55c52d32d058a5284c4eec485a972bf3d70ebaea540204bf9f9e5024cd" exitCode=0 Jul 09 19:46:03.667076 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.666729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" event={"ID":"f349fa74-6d5b-4d91-aeec-6acf384487c6","Type":"ContainerDied","Data":"53d0fe55c52d32d058a5284c4eec485a972bf3d70ebaea540204bf9f9e5024cd"} Jul 09 19:46:03.668215 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.668189 2562 generic.go:356] "Generic (PLEG): container finished" podID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerID="2a22711893910729beac2e9590b446278386daaf6d77dd34d504860b2a36f9c2" exitCode=0 Jul 09 19:46:03.668324 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.668270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" event={"ID":"98a4ed1a-9efd-4976-8732-799ab79c5e6b","Type":"ContainerDied","Data":"2a22711893910729beac2e9590b446278386daaf6d77dd34d504860b2a36f9c2"} Jul 09 19:46:03.668324 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.668313 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" event={"ID":"98a4ed1a-9efd-4976-8732-799ab79c5e6b","Type":"ContainerStarted","Data":"c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192"} Jul 09 19:46:03.670147 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.670121 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a5a4166-2d55-414f-a876-28017499e692" containerID="a50e00de9a242e1f0fd8acff396e2dd1aa25cd41ed90695951bd923439c31d20" exitCode=0 Jul 09 19:46:03.670266 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:03.670150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" event={"ID":"0a5a4166-2d55-414f-a876-28017499e692","Type":"ContainerDied","Data":"a50e00de9a242e1f0fd8acff396e2dd1aa25cd41ed90695951bd923439c31d20"} Jul 09 19:46:04.677670 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.677639 2562 generic.go:356] "Generic (PLEG): container finished" podID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerID="f8bf8a2cb4c6e4cad9d7f166aa2fd9db3fbf8f654ba7d1aa3e81e374733836f7" exitCode=0 Jul 09 19:46:04.678117 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.677729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" event={"ID":"98a4ed1a-9efd-4976-8732-799ab79c5e6b","Type":"ContainerDied","Data":"f8bf8a2cb4c6e4cad9d7f166aa2fd9db3fbf8f654ba7d1aa3e81e374733836f7"} Jul 09 19:46:04.680042 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.680022 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a5a4166-2d55-414f-a876-28017499e692" containerID="28abe41e990fd44cac285058052322d55aa94b9cd8e49c3fd9cc986bc4dc57bf" exitCode=0 Jul 09 19:46:04.680149 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.680088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" event={"ID":"0a5a4166-2d55-414f-a876-28017499e692","Type":"ContainerDied","Data":"28abe41e990fd44cac285058052322d55aa94b9cd8e49c3fd9cc986bc4dc57bf"} Jul 09 19:46:04.681969 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.681946 2562 generic.go:356] "Generic (PLEG): container finished" podID="7697830a-cdf4-42bb-839c-699a74718ce5" containerID="ef7554a091e4eecb370e402b74485ea3f9765f9b5ba8775bc1612d4a1136ab2a" exitCode=0 Jul 09 19:46:04.682066 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.682028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" event={"ID":"7697830a-cdf4-42bb-839c-699a74718ce5","Type":"ContainerDied","Data":"ef7554a091e4eecb370e402b74485ea3f9765f9b5ba8775bc1612d4a1136ab2a"} Jul 09 19:46:04.820422 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.820384 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:05.000051 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:04.999970 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle\") pod \"f349fa74-6d5b-4d91-aeec-6acf384487c6\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " Jul 09 19:46:05.000051 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.000021 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util\") pod \"f349fa74-6d5b-4d91-aeec-6acf384487c6\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " Jul 09 19:46:05.000220 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.000066 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2h2b\" (UniqueName: \"kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b\") pod \"f349fa74-6d5b-4d91-aeec-6acf384487c6\" (UID: \"f349fa74-6d5b-4d91-aeec-6acf384487c6\") " Jul 09 19:46:05.000537 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.000499 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle" (OuterVolumeSpecName: "bundle") pod "f349fa74-6d5b-4d91-aeec-6acf384487c6" (UID: "f349fa74-6d5b-4d91-aeec-6acf384487c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:05.002351 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.002326 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b" (OuterVolumeSpecName: "kube-api-access-r2h2b") pod "f349fa74-6d5b-4d91-aeec-6acf384487c6" (UID: "f349fa74-6d5b-4d91-aeec-6acf384487c6"). InnerVolumeSpecName "kube-api-access-r2h2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:46:05.005627 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.005605 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util" (OuterVolumeSpecName: "util") pod "f349fa74-6d5b-4d91-aeec-6acf384487c6" (UID: "f349fa74-6d5b-4d91-aeec-6acf384487c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:05.101205 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.101168 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:05.101205 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.101207 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2h2b\" (UniqueName: \"kubernetes.io/projected/f349fa74-6d5b-4d91-aeec-6acf384487c6-kube-api-access-r2h2b\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:05.101449 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.101222 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f349fa74-6d5b-4d91-aeec-6acf384487c6-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:05.688769 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.688739 2562 generic.go:356] "Generic (PLEG): container finished" podID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerID="52910e81e7046a0c2558758883ce6c565ad1a206ba9218522dcaddbb5caa2bc7" exitCode=0 Jul 09 19:46:05.689222 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.688819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" event={"ID":"98a4ed1a-9efd-4976-8732-799ab79c5e6b","Type":"ContainerDied","Data":"52910e81e7046a0c2558758883ce6c565ad1a206ba9218522dcaddbb5caa2bc7"} Jul 09 19:46:05.690937 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.690910 2562 generic.go:356] "Generic (PLEG): container finished" podID="7697830a-cdf4-42bb-839c-699a74718ce5" containerID="9c387d969962fd7f30d47789d531f173a15e70fdbd83dbc09ebddcb61175913b" exitCode=0 Jul 09 19:46:05.691074 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.690988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" event={"ID":"7697830a-cdf4-42bb-839c-699a74718ce5","Type":"ContainerDied","Data":"9c387d969962fd7f30d47789d531f173a15e70fdbd83dbc09ebddcb61175913b"} Jul 09 19:46:05.692740 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.692718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" event={"ID":"f349fa74-6d5b-4d91-aeec-6acf384487c6","Type":"ContainerDied","Data":"02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919"} Jul 09 19:46:05.692832 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.692763 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b3ef75ef477596967bb3a8110606766608e8ebfd9a8514d345bdaae5e3c919" Jul 09 19:46:05.692956 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.692941 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b" Jul 09 19:46:05.822084 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:05.822057 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:06.010086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.010011 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk7vl\" (UniqueName: \"kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl\") pod \"0a5a4166-2d55-414f-a876-28017499e692\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " Jul 09 19:46:06.010086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.010064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle\") pod \"0a5a4166-2d55-414f-a876-28017499e692\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " Jul 09 19:46:06.010322 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.010108 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util\") pod \"0a5a4166-2d55-414f-a876-28017499e692\" (UID: \"0a5a4166-2d55-414f-a876-28017499e692\") " Jul 09 19:46:06.010753 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.010725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle" (OuterVolumeSpecName: "bundle") pod "0a5a4166-2d55-414f-a876-28017499e692" (UID: "0a5a4166-2d55-414f-a876-28017499e692"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:06.012334 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.012314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl" (OuterVolumeSpecName: "kube-api-access-bk7vl") pod "0a5a4166-2d55-414f-a876-28017499e692" (UID: "0a5a4166-2d55-414f-a876-28017499e692"). InnerVolumeSpecName "kube-api-access-bk7vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:46:06.015803 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.015783 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util" (OuterVolumeSpecName: "util") pod "0a5a4166-2d55-414f-a876-28017499e692" (UID: "0a5a4166-2d55-414f-a876-28017499e692"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:06.111118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.111079 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7vl\" (UniqueName: \"kubernetes.io/projected/0a5a4166-2d55-414f-a876-28017499e692-kube-api-access-bk7vl\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:06.111118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.111114 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:06.111118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.111124 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a5a4166-2d55-414f-a876-28017499e692-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:06.700752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.700722 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" Jul 09 19:46:06.700752 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.700747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm" event={"ID":"0a5a4166-2d55-414f-a876-28017499e692","Type":"ContainerDied","Data":"2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216"} Jul 09 19:46:06.701185 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.700802 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d43aa72f2b936b738e78a1cb8b335dc80c850ae457d47d1030eed638c205216" Jul 09 19:46:06.853563 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.853537 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:06.871983 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:06.871962 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:07.018086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018005 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle\") pod \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " Jul 09 19:46:07.018086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018042 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util\") pod \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " Jul 09 19:46:07.018086 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018066 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lk2k\" (UniqueName: \"kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k\") pod \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\" (UID: \"98a4ed1a-9efd-4976-8732-799ab79c5e6b\") " Jul 09 19:46:07.018353 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018095 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle\") pod \"7697830a-cdf4-42bb-839c-699a74718ce5\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " Jul 09 19:46:07.018353 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018128 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util\") pod \"7697830a-cdf4-42bb-839c-699a74718ce5\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " Jul 09 19:46:07.018353 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018165 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxt4n\" (UniqueName: \"kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n\") pod \"7697830a-cdf4-42bb-839c-699a74718ce5\" (UID: \"7697830a-cdf4-42bb-839c-699a74718ce5\") " Jul 09 19:46:07.018717 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.018685 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle" (OuterVolumeSpecName: "bundle") pod "98a4ed1a-9efd-4976-8732-799ab79c5e6b" (UID: "98a4ed1a-9efd-4976-8732-799ab79c5e6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:07.019100 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.019072 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle" (OuterVolumeSpecName: "bundle") pod "7697830a-cdf4-42bb-839c-699a74718ce5" (UID: "7697830a-cdf4-42bb-839c-699a74718ce5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:07.020382 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.020354 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k" (OuterVolumeSpecName: "kube-api-access-9lk2k") pod "98a4ed1a-9efd-4976-8732-799ab79c5e6b" (UID: "98a4ed1a-9efd-4976-8732-799ab79c5e6b"). InnerVolumeSpecName "kube-api-access-9lk2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:46:07.020463 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.020394 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n" (OuterVolumeSpecName: "kube-api-access-gxt4n") pod "7697830a-cdf4-42bb-839c-699a74718ce5" (UID: "7697830a-cdf4-42bb-839c-699a74718ce5"). InnerVolumeSpecName "kube-api-access-gxt4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:46:07.024193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.024157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util" (OuterVolumeSpecName: "util") pod "7697830a-cdf4-42bb-839c-699a74718ce5" (UID: "7697830a-cdf4-42bb-839c-699a74718ce5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:07.024256 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.024238 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util" (OuterVolumeSpecName: "util") pod "98a4ed1a-9efd-4976-8732-799ab79c5e6b" (UID: "98a4ed1a-9efd-4976-8732-799ab79c5e6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:46:07.119491 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119464 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.119491 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119488 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98a4ed1a-9efd-4976-8732-799ab79c5e6b-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.119635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119496 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lk2k\" (UniqueName: \"kubernetes.io/projected/98a4ed1a-9efd-4976-8732-799ab79c5e6b-kube-api-access-9lk2k\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.119635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119507 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-bundle\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.119635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119515 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7697830a-cdf4-42bb-839c-699a74718ce5-util\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.119635 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.119523 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxt4n\" (UniqueName: \"kubernetes.io/projected/7697830a-cdf4-42bb-839c-699a74718ce5-kube-api-access-gxt4n\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:46:07.707786 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.707757 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" Jul 09 19:46:07.708235 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.707784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m" event={"ID":"98a4ed1a-9efd-4976-8732-799ab79c5e6b","Type":"ContainerDied","Data":"c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192"} Jul 09 19:46:07.708235 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.707835 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19fb5f6176e0867c76fd6d161d0b65c720f1b354aaf61cc0b9c077927770192" Jul 09 19:46:07.709516 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.709489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" event={"ID":"7697830a-cdf4-42bb-839c-699a74718ce5","Type":"ContainerDied","Data":"a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3"} Jul 09 19:46:07.709689 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.709526 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57b30e3372176959731c68ebcb85aa85a383241876b210ba7fd4ef6235346c3" Jul 09 19:46:07.709689 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:07.709496 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6" Jul 09 19:46:12.279618 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:12.279512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:12.348336 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:12.348306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:14.687612 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:14.687555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:17.226397 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226362 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2"] Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226672 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226684 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226695 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226701 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226709 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226714 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226721 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226726 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226732 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226737 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226744 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226749 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226755 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226760 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226766 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226771 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226777 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226782 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226792 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226796 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226802 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226807 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="util" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226817 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226822 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="pull" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226868 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7697830a-cdf4-42bb-839c-699a74718ce5" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226876 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a4ed1a-9efd-4976-8732-799ab79c5e6b" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226881 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5a4166-2d55-414f-a876-28017499e692" containerName="extract" Jul 09 19:46:17.226910 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.226888 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f349fa74-6d5b-4d91-aeec-6acf384487c6" containerName="extract" Jul 09 19:46:17.230887 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.230871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:17.236369 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.236346 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fh8d5\"" Jul 09 19:46:17.245970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.245945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2"] Jul 09 19:46:17.299729 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.299698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6z8\" (UniqueName: \"kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8\") pod \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" (UID: \"d6c78fef-1c32-4323-a44a-8437be07415f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:17.400136 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.400102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6z8\" (UniqueName: \"kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8\") pod \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" (UID: \"d6c78fef-1c32-4323-a44a-8437be07415f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:17.413448 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.413425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6z8\" (UniqueName: \"kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8\") pod \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" (UID: \"d6c78fef-1c32-4323-a44a-8437be07415f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:17.541299 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.541217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:17.679978 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.679953 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2"] Jul 09 19:46:17.681713 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:17.681686 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c78fef_1c32_4323_a44a_8437be07415f.slice/crio-6e82c4d073fd6d0bb4f805cef82bb18aa18fdd4eaee4205721913c7281d1c9ea WatchSource:0}: Error finding container 6e82c4d073fd6d0bb4f805cef82bb18aa18fdd4eaee4205721913c7281d1c9ea: Status 404 returned error can't find the container with id 6e82c4d073fd6d0bb4f805cef82bb18aa18fdd4eaee4205721913c7281d1c9ea Jul 09 19:46:17.757809 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:17.757777 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" event={"ID":"d6c78fef-1c32-4323-a44a-8437be07415f","Type":"ContainerStarted","Data":"6e82c4d073fd6d0bb4f805cef82bb18aa18fdd4eaee4205721913c7281d1c9ea"} Jul 09 19:46:18.794824 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:18.794780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:20.682716 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.682682 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w"] Jul 09 19:46:20.689970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.689951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:20.693038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.693015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 19:46:20.693191 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.693104 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5mh2q\"" Jul 09 19:46:20.697221 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.697201 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w"] Jul 09 19:46:20.724543 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.724510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87jj\" (UniqueName: \"kubernetes.io/projected/b56839c4-2f35-40bf-a73f-5396bea70761-kube-api-access-k87jj\") pod \"dns-operator-controller-manager-864dd78c4c-g6n9w\" (UID: \"b56839c4-2f35-40bf-a73f-5396bea70761\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:20.776719 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.776683 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" event={"ID":"d6c78fef-1c32-4323-a44a-8437be07415f","Type":"ContainerStarted","Data":"98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7"} Jul 09 19:46:20.776909 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.776815 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:20.803142 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.800898 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" podStartSLOduration=1.5084782730000001 podStartE2EDuration="3.800875862s" totalImagesPullingTime="2.292397589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:46:20.79687345 +0000 UTC m=+407.387633655" watchObservedRunningTime="2026-07-09 19:46:20.800875862 +0000 UTC m=+407.391636069" Jul 09 19:46:20.825570 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.825540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87jj\" (UniqueName: \"kubernetes.io/projected/b56839c4-2f35-40bf-a73f-5396bea70761-kube-api-access-k87jj\") pod \"dns-operator-controller-manager-864dd78c4c-g6n9w\" (UID: \"b56839c4-2f35-40bf-a73f-5396bea70761\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:20.836324 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:20.836295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87jj\" (UniqueName: \"kubernetes.io/projected/b56839c4-2f35-40bf-a73f-5396bea70761-kube-api-access-k87jj\") pod \"dns-operator-controller-manager-864dd78c4c-g6n9w\" (UID: \"b56839c4-2f35-40bf-a73f-5396bea70761\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:21.000702 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:21.000604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:21.138060 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:21.138035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w"] Jul 09 19:46:21.139744 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:21.139713 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56839c4_2f35_40bf_a73f_5396bea70761.slice/crio-630e200ad87e2f48150f36a432c9606d8d08e2bf2f9f45f5e3c4d2d882ffd1bd WatchSource:0}: Error finding container 630e200ad87e2f48150f36a432c9606d8d08e2bf2f9f45f5e3c4d2d882ffd1bd: Status 404 returned error can't find the container with id 630e200ad87e2f48150f36a432c9606d8d08e2bf2f9f45f5e3c4d2d882ffd1bd Jul 09 19:46:21.157849 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:21.157820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:21.761940 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:21.761901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:21.782711 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:21.782681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" event={"ID":"b56839c4-2f35-40bf-a73f-5396bea70761","Type":"ContainerStarted","Data":"630e200ad87e2f48150f36a432c9606d8d08e2bf2f9f45f5e3c4d2d882ffd1bd"} Jul 09 19:46:22.033808 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:22.033702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.183456 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.183422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.285514 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.285476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.385758 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.385717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.484132 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.484031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.792193 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.792070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:23.795306 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.795276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" event={"ID":"b56839c4-2f35-40bf-a73f-5396bea70761","Type":"ContainerStarted","Data":"c392aecda02948e2f73856ea22fef62eb2aa0e600512bcb88d7866a348453007"} Jul 09 19:46:23.795472 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.795404 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:23.816203 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:23.816140 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" podStartSLOduration=1.424762136 podStartE2EDuration="3.81612247s" totalImagesPullingTime="2.391360334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:46:23.814007456 +0000 UTC m=+410.404767675" watchObservedRunningTime="2026-07-09 19:46:23.81612247 +0000 UTC m=+410.406882678" Jul 09 19:46:24.448162 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.448128 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7"] Jul 09 19:46:24.451360 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.451344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:24.454199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.454183 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4cbqk\"" Jul 09 19:46:24.464774 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.464748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7"] Jul 09 19:46:24.557223 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.557176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9v6\" (UniqueName: \"kubernetes.io/projected/4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d-kube-api-access-lm9v6\") pod \"authorino-operator-75cc8d5cb7-wqjh7\" (UID: \"4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:24.657771 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.657720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9v6\" (UniqueName: \"kubernetes.io/projected/4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d-kube-api-access-lm9v6\") pod \"authorino-operator-75cc8d5cb7-wqjh7\" (UID: \"4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:24.674179 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.674143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9v6\" (UniqueName: \"kubernetes.io/projected/4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d-kube-api-access-lm9v6\") pod \"authorino-operator-75cc8d5cb7-wqjh7\" (UID: \"4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:24.762080 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.761978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:24.900463 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:24.900439 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7"] Jul 09 19:46:24.902562 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:24.902524 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4758cc9a_897f_4f09_b6d1_c32b3ba0bc5d.slice/crio-e74e2005e1b2a21e5f754a346147dc92f79fda3a7a8486c3c9a8fc5734c857ce WatchSource:0}: Error finding container e74e2005e1b2a21e5f754a346147dc92f79fda3a7a8486c3c9a8fc5734c857ce: Status 404 returned error can't find the container with id e74e2005e1b2a21e5f754a346147dc92f79fda3a7a8486c3c9a8fc5734c857ce Jul 09 19:46:25.177885 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:25.177855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:25.270958 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:25.270921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:25.808822 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:25.808783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" event={"ID":"4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d","Type":"ContainerStarted","Data":"e74e2005e1b2a21e5f754a346147dc92f79fda3a7a8486c3c9a8fc5734c857ce"} Jul 09 19:46:27.031071 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:27.031032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:27.822326 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:27.822284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" event={"ID":"4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d","Type":"ContainerStarted","Data":"256a2829784a7188eee44ad2d52d689ed4b1058d275c6003c21089ccd4b41edf"} Jul 09 19:46:27.822525 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:27.822363 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:27.844318 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:27.844265 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" podStartSLOduration=1.51798259 podStartE2EDuration="3.844250213s" totalImagesPullingTime="2.326267623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:46:27.841946485 +0000 UTC m=+414.432706704" watchObservedRunningTime="2026-07-09 19:46:27.844250213 +0000 UTC m=+414.435010417" Jul 09 19:46:29.118607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.118546 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs"] Jul 09 19:46:29.121855 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.121830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.125784 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.125763 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jvqwt\"" Jul 09 19:46:29.141152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.141112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs"] Jul 09 19:46:29.197443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.197403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.197621 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.197460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mk9\" (UniqueName: \"kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.298443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.298406 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mk9\" (UniqueName: \"kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.298640 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.298486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.298924 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.298905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.309306 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.309285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mk9\" (UniqueName: \"kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.432072 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.432036 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:29.593820 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.593793 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs"] Jul 09 19:46:29.594970 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:29.594944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0989b43_1948_47bd_826d_b1bd63a34abf.slice/crio-04d9c138c32030a8781cfbb78c798b6c3373b329f4e2c1bf738c9a96c42b4270 WatchSource:0}: Error finding container 04d9c138c32030a8781cfbb78c798b6c3373b329f4e2c1bf738c9a96c42b4270: Status 404 returned error can't find the container with id 04d9c138c32030a8781cfbb78c798b6c3373b329f4e2c1bf738c9a96c42b4270 Jul 09 19:46:29.832971 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:29.832883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" event={"ID":"d0989b43-1948-47bd-826d-b1bd63a34abf","Type":"ContainerStarted","Data":"04d9c138c32030a8781cfbb78c798b6c3373b329f4e2c1bf738c9a96c42b4270"} Jul 09 19:46:30.201715 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:30.201663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:30.308510 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:30.308478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:31.785718 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:31.785686 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:33.589994 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:33.589961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:33.692323 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:33.692288 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:34.803318 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:34.803283 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-g6n9w" Jul 09 19:46:35.868557 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:35.868525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" event={"ID":"d0989b43-1948-47bd-826d-b1bd63a34abf","Type":"ContainerStarted","Data":"671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a"} Jul 09 19:46:35.868942 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:35.868662 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:35.900741 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:35.900681 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" podStartSLOduration=0.772752825 podStartE2EDuration="6.900666005s" totalImagesPullingTime="6.12791318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:46:35.898792626 +0000 UTC m=+422.489552834" watchObservedRunningTime="2026-07-09 19:46:35.900666005 +0000 UTC m=+422.491426211" Jul 09 19:46:38.377661 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.377628 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp"] Jul 09 19:46:38.382168 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.382142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:38.382598 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.382561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.385448 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.385428 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 19:46:38.385991 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.385974 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 19:46:38.386074 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.386014 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62ctk\"" Jul 09 19:46:38.391026 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.391006 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp"] Jul 09 19:46:38.477163 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.477124 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfj8\" (UniqueName: \"kubernetes.io/projected/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-kube-api-access-psfj8\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.477163 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.477167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.477372 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.477201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.578506 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.578475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfj8\" (UniqueName: \"kubernetes.io/projected/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-kube-api-access-psfj8\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.578506 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.578507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.578724 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.578531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.578724 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:38.578644 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 09 19:46:38.578724 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:38.578704 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert podName:a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42 nodeName:}" failed. No retries permitted until 2026-07-09 19:46:39.078689169 +0000 UTC m=+425.669449352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-8l4hp" (UID: "a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42") : secret "plugin-serving-cert" not found Jul 09 19:46:38.579173 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.579155 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.588232 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.588211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfj8\" (UniqueName: \"kubernetes.io/projected/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-kube-api-access-psfj8\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:38.816174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.816138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:38.829701 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:38.829680 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wqjh7" Jul 09 19:46:39.082569 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.082470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:39.085107 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.085082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8l4hp\" (UID: \"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:39.292715 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.292676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" Jul 09 19:46:39.422694 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.422659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:39.443353 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.443333 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp"] Jul 09 19:46:39.445369 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:46:39.445343 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda403c0b2_4f9b_42b8_bcd3_cc5049f9ac42.slice/crio-cce9d598cb0874f95ab49bfb1c7e03954587870496591ba7c9f4ae1764c005e9 WatchSource:0}: Error finding container cce9d598cb0874f95ab49bfb1c7e03954587870496591ba7c9f4ae1764c005e9: Status 404 returned error can't find the container with id cce9d598cb0874f95ab49bfb1c7e03954587870496591ba7c9f4ae1764c005e9 Jul 09 19:46:39.517733 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.517704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:39.628689 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.628574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:39.782943 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.782905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:39.883308 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.883211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:39.889719 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.889691 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" event={"ID":"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42","Type":"ContainerStarted","Data":"cce9d598cb0874f95ab49bfb1c7e03954587870496591ba7c9f4ae1764c005e9"} Jul 09 19:46:39.983373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:39.983346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:41.798003 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:41.797961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:41.900256 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:41.900219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:42.771201 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:42.771056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:42.870318 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:42.870230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:46.876393 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:46.876360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:48.730662 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.730627 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs"] Jul 09 19:46:48.731260 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.730873 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" containerName="manager" containerID="cri-o://671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a" gracePeriod=2 Jul 09 19:46:48.746147 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.746097 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs"] Jul 09 19:46:48.751380 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.751336 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d0989b43-1948-47bd-826d-b1bd63a34abf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-trwvs\": pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:48.763008 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.762981 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2"] Jul 09 19:46:48.763234 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.763210 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" containerName="manager" containerID="cri-o://98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7" gracePeriod=2 Jul 09 19:46:48.766169 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.766126 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:48.773157 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.773124 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:46:48.773562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.773546 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" containerName="manager" Jul 09 19:46:48.773562 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.773563 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" containerName="manager" Jul 09 19:46:48.773776 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.773668 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" containerName="manager" Jul 09 19:46:48.776873 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.776855 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.780314 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.780279 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d6c78fef-1c32-4323-a44a-8437be07415f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-5f8c2\": pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:48.781195 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.781168 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2"] Jul 09 19:46:48.784293 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.784257 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:48.786612 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.786566 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:48.792016 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.791931 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:46:48.797442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.796492 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt"] Jul 09 19:46:48.797442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.796975 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" containerName="manager" Jul 09 19:46:48.797442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.796991 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" containerName="manager" Jul 09 19:46:48.797442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.797081 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" containerName="manager" Jul 09 19:46:48.800432 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.800415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:46:48.809507 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.809479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt"] Jul 09 19:46:48.843532 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.843491 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:46:48.845714 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:46:48.845679 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:46:48.873077 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.873041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmc6\" (UniqueName: \"kubernetes.io/projected/783925ea-937f-4803-9c6a-c4796a98a073-kube-api-access-zmmc6\") pod \"limitador-operator-controller-manager-859d7b78bd-zqmlt\" (UID: \"783925ea-937f-4803-9c6a-c4796a98a073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:46:48.873255 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.873134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.873255 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.873178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtqh\" (UniqueName: \"kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.974346 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.974297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmc6\" (UniqueName: \"kubernetes.io/projected/783925ea-937f-4803-9c6a-c4796a98a073-kube-api-access-zmmc6\") pod \"limitador-operator-controller-manager-859d7b78bd-zqmlt\" (UID: \"783925ea-937f-4803-9c6a-c4796a98a073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:46:48.974551 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.974383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.974551 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.974409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtqh\" (UniqueName: \"kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.974939 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.974917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:48.987335 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.987275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmc6\" (UniqueName: \"kubernetes.io/projected/783925ea-937f-4803-9c6a-c4796a98a073-kube-api-access-zmmc6\") pod \"limitador-operator-controller-manager-859d7b78bd-zqmlt\" (UID: \"783925ea-937f-4803-9c6a-c4796a98a073\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:46:48.987681 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:48.987658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtqh\" (UniqueName: \"kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bnzbq\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:49.139073 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:49.139039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:46:49.147706 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:49.147682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:46:49.497331 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:49.497296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:49.598913 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:49.598862 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:52.422776 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:52.422740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:52.527013 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:52.526944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:57.230860 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:57.230644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:57.328223 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:57.328187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:57.428307 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:57.428242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:46:57.501295 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:46:57.501210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:02.406063 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.405991 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:02.408658 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:02.408575 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:02.409024 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.409002 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt"] Jul 09 19:47:02.410306 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:47:02.410269 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783925ea_937f_4803_9c6a_c4796a98a073.slice/crio-2fd0f6787275572ab95124f5ed96a85054af8b0430dcb4602fdcf3b5db06d53f WatchSource:0}: Error finding container 2fd0f6787275572ab95124f5ed96a85054af8b0430dcb4602fdcf3b5db06d53f: Status 404 returned error can't find the container with id 2fd0f6787275572ab95124f5ed96a85054af8b0430dcb4602fdcf3b5db06d53f Jul 09 19:47:02.416817 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.416795 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:02.419242 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:02.419216 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:02.422704 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:02.422671 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:02.438788 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.438766 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:47:02.441866 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:47:02.441836 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5634de_566e_4627_b1a2_6ee761d4bcbd.slice/crio-2a00e97d3feaf52e58d56b97a4664186fb1ca50a3681133dcacfbd968143cb88 WatchSource:0}: Error finding container 2a00e97d3feaf52e58d56b97a4664186fb1ca50a3681133dcacfbd968143cb88: Status 404 returned error can't find the container with id 2a00e97d3feaf52e58d56b97a4664186fb1ca50a3681133dcacfbd968143cb88 Jul 09 19:47:02.493800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.493777 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49mk9\" (UniqueName: \"kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9\") pod \"d0989b43-1948-47bd-826d-b1bd63a34abf\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " Jul 09 19:47:02.493905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.493857 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw6z8\" (UniqueName: \"kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8\") pod \"d6c78fef-1c32-4323-a44a-8437be07415f\" (UID: \"d6c78fef-1c32-4323-a44a-8437be07415f\") " Jul 09 19:47:02.493905 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.493895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume\") pod \"d0989b43-1948-47bd-826d-b1bd63a34abf\" (UID: \"d0989b43-1948-47bd-826d-b1bd63a34abf\") " Jul 09 19:47:02.494450 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.494352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d0989b43-1948-47bd-826d-b1bd63a34abf" (UID: "d0989b43-1948-47bd-826d-b1bd63a34abf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:47:02.495856 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.495790 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9" (OuterVolumeSpecName: "kube-api-access-49mk9") pod "d0989b43-1948-47bd-826d-b1bd63a34abf" (UID: "d0989b43-1948-47bd-826d-b1bd63a34abf"). InnerVolumeSpecName "kube-api-access-49mk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:47:02.495930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.495910 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8" (OuterVolumeSpecName: "kube-api-access-kw6z8") pod "d6c78fef-1c32-4323-a44a-8437be07415f" (UID: "d6c78fef-1c32-4323-a44a-8437be07415f"). InnerVolumeSpecName "kube-api-access-kw6z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:47:02.595013 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.594964 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0989b43-1948-47bd-826d-b1bd63a34abf-extensions-socket-volume\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:02.595013 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.595009 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49mk9\" (UniqueName: \"kubernetes.io/projected/d0989b43-1948-47bd-826d-b1bd63a34abf-kube-api-access-49mk9\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:02.595013 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.595020 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6z8\" (UniqueName: \"kubernetes.io/projected/d6c78fef-1c32-4323-a44a-8437be07415f-kube-api-access-kw6z8\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:02.639174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.639137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:02.738878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:02.738776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:03.038020 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.037931 2562 generic.go:356] "Generic (PLEG): container finished" podID="d6c78fef-1c32-4323-a44a-8437be07415f" containerID="98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7" exitCode=0 Jul 09 19:47:03.038020 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.037973 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:03.038459 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.038025 2562 scope.go:117] "RemoveContainer" containerID="98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7" Jul 09 19:47:03.039687 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.039647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" event={"ID":"a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42","Type":"ContainerStarted","Data":"6111819d4b60b8f971506c55164101e9d70324e986de9427dd2a3001a70ed688"} Jul 09 19:47:03.041638 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.041614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" event={"ID":"783925ea-937f-4803-9c6a-c4796a98a073","Type":"ContainerStarted","Data":"dcc44d0251100393f1f180920a14a361fecdbaa02cf78e9667e429107cc3dc37"} Jul 09 19:47:03.041765 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.041660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" event={"ID":"783925ea-937f-4803-9c6a-c4796a98a073","Type":"ContainerStarted","Data":"2fd0f6787275572ab95124f5ed96a85054af8b0430dcb4602fdcf3b5db06d53f"} Jul 09 19:47:03.041765 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.041691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:47:03.042084 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.042054 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:03.043536 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.043509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" event={"ID":"de5634de-566e-4627-b1a2-6ee761d4bcbd","Type":"ContainerStarted","Data":"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9"} Jul 09 19:47:03.043679 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.043568 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:47:03.043679 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.043616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" event={"ID":"de5634de-566e-4627-b1a2-6ee761d4bcbd","Type":"ContainerStarted","Data":"2a00e97d3feaf52e58d56b97a4664186fb1ca50a3681133dcacfbd968143cb88"} Jul 09 19:47:03.044961 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.044940 2562 generic.go:356] "Generic (PLEG): container finished" podID="d0989b43-1948-47bd-826d-b1bd63a34abf" containerID="671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a" exitCode=0 Jul 09 19:47:03.045051 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.044995 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:03.046093 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.046064 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:03.048286 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.048265 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:03.048489 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.048463 2562 scope.go:117] "RemoveContainer" containerID="98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7" Jul 09 19:47:03.048788 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.048768 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7\": container with ID starting with 98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7 not found: ID does not exist" containerID="98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7" Jul 09 19:47:03.048867 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.048795 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7"} err="failed to get container status \"98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7\": rpc error: code = NotFound desc = could not find container \"98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7\": container with ID starting with 98122e3ca3b177770ecd02a2602a0f90b98c8d3d6d693e87ee5f8fa0c2b7f9e7 not found: ID does not exist" Jul 09 19:47:03.048867 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.048812 2562 scope.go:117] "RemoveContainer" containerID="671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a" Jul 09 19:47:03.058338 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.058321 2562 scope.go:117] "RemoveContainer" containerID="671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a" Jul 09 19:47:03.058632 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.058614 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a\": container with ID starting with 671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a not found: ID does not exist" containerID="671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a" Jul 09 19:47:03.058685 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.058641 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a"} err="failed to get container status \"671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a\": rpc error: code = NotFound desc = could not find container \"671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a\": container with ID starting with 671e43e53cbb163039b8edefdd8e504e586edc5a28ec64d2805dd8cd95414d6a not found: ID does not exist" Jul 09 19:47:03.077381 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.077329 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8l4hp" podStartSLOduration=2.182520481 podStartE2EDuration="25.077315104s" totalImagesPullingTime="22.894794623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:03.076146881 +0000 UTC m=+449.666907086" watchObservedRunningTime="2026-07-09 19:47:03.077315104 +0000 UTC m=+449.668075308" Jul 09 19:47:03.114616 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.114545 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" podStartSLOduration=15.114528041 podStartE2EDuration="15.114528041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:03.113863313 +0000 UTC m=+449.704623519" watchObservedRunningTime="2026-07-09 19:47:03.114528041 +0000 UTC m=+449.705288248" Jul 09 19:47:03.118353 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.118321 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:03.144769 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.144701 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" podStartSLOduration=15.144678871 podStartE2EDuration="15.144678871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:03.143909192 +0000 UTC m=+449.734669413" watchObservedRunningTime="2026-07-09 19:47:03.144678871 +0000 UTC m=+449.735439080" Jul 09 19:47:03.147291 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.147257 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:03.149468 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.149447 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:03.953189 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.953154 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" path="/var/lib/kubelet/pods/d0989b43-1948-47bd-826d-b1bd63a34abf/volumes" Jul 09 19:47:03.953558 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:03.953468 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" path="/var/lib/kubelet/pods/d6c78fef-1c32-4323-a44a-8437be07415f/volumes" Jul 09 19:47:03.953634 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.953566 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5f8c2\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d6c78fef-1c32-4323-a44a-8437be07415f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5f8c2" Jul 09 19:47:03.957174 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:03.957152 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-trwvs\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d0989b43-1948-47bd-826d-b1bd63a34abf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-trwvs" Jul 09 19:47:06.216512 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.216480 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:47:06.217055 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.216671 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" podUID="de5634de-566e-4627-b1a2-6ee761d4bcbd" containerName="manager" containerID="cri-o://3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9" gracePeriod=10 Jul 09 19:47:06.469813 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.469755 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:47:06.628993 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.628964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txtqh\" (UniqueName: \"kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh\") pod \"de5634de-566e-4627-b1a2-6ee761d4bcbd\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " Jul 09 19:47:06.629152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.629063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume\") pod \"de5634de-566e-4627-b1a2-6ee761d4bcbd\" (UID: \"de5634de-566e-4627-b1a2-6ee761d4bcbd\") " Jul 09 19:47:06.629459 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.629434 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "de5634de-566e-4627-b1a2-6ee761d4bcbd" (UID: "de5634de-566e-4627-b1a2-6ee761d4bcbd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:47:06.631109 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.631085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh" (OuterVolumeSpecName: "kube-api-access-txtqh") pod "de5634de-566e-4627-b1a2-6ee761d4bcbd" (UID: "de5634de-566e-4627-b1a2-6ee761d4bcbd"). InnerVolumeSpecName "kube-api-access-txtqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:47:06.730344 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.730264 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txtqh\" (UniqueName: \"kubernetes.io/projected/de5634de-566e-4627-b1a2-6ee761d4bcbd-kube-api-access-txtqh\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:06.730344 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:06.730287 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de5634de-566e-4627-b1a2-6ee761d4bcbd-extensions-socket-volume\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:07.067765 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.067665 2562 generic.go:356] "Generic (PLEG): container finished" podID="de5634de-566e-4627-b1a2-6ee761d4bcbd" containerID="3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9" exitCode=0 Jul 09 19:47:07.067765 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.067720 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" Jul 09 19:47:07.067765 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.067747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" event={"ID":"de5634de-566e-4627-b1a2-6ee761d4bcbd","Type":"ContainerDied","Data":"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9"} Jul 09 19:47:07.068041 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.067801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq" event={"ID":"de5634de-566e-4627-b1a2-6ee761d4bcbd","Type":"ContainerDied","Data":"2a00e97d3feaf52e58d56b97a4664186fb1ca50a3681133dcacfbd968143cb88"} Jul 09 19:47:07.068041 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.067823 2562 scope.go:117] "RemoveContainer" containerID="3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9" Jul 09 19:47:07.077599 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.077562 2562 scope.go:117] "RemoveContainer" containerID="3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9" Jul 09 19:47:07.077848 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:07.077832 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9\": container with ID starting with 3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9 not found: ID does not exist" containerID="3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9" Jul 09 19:47:07.077915 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.077863 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9"} err="failed to get container status \"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9\": rpc error: code = NotFound desc = could not find container \"3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9\": container with ID starting with 3674b234cbfd721ef45c5d60b4675b1e9bf8148b25ee1d7b952a6ae57dcf13a9 not found: ID does not exist" Jul 09 19:47:07.090850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.090823 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:47:07.100883 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.100859 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bnzbq"] Jul 09 19:47:07.953055 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:07.953024 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5634de-566e-4627-b1a2-6ee761d4bcbd" path="/var/lib/kubelet/pods/de5634de-566e-4627-b1a2-6ee761d4bcbd/volumes" Jul 09 19:47:14.053539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:14.053510 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zqmlt" Jul 09 19:47:22.709989 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.709957 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q"] Jul 09 19:47:22.710442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.710296 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5634de-566e-4627-b1a2-6ee761d4bcbd" containerName="manager" Jul 09 19:47:22.710442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.710307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5634de-566e-4627-b1a2-6ee761d4bcbd" containerName="manager" Jul 09 19:47:22.710442 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.710377 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5634de-566e-4627-b1a2-6ee761d4bcbd" containerName="manager" Jul 09 19:47:22.731699 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.731664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:22.867243 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.867211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q"] Jul 09 19:47:22.867423 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.867332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.870757 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.870737 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q2kqt\"" Jul 09 19:47:22.954678 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.954678 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.954878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.954878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fb299f-3f49-4a3d-9631-99d70d8dc040-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.954878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.954878 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xftm\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-kube-api-access-6xftm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.955049 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.955049 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.955049 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.954971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:22.973818 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:22.973744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:23.055623 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fb299f-3f49-4a3d-9631-99d70d8dc040-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xftm\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-kube-api-access-6xftm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.055800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.055773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056151 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056151 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056151 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056297 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056355 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056446 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.056565 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.056549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fb299f-3f49-4a3d-9631-99d70d8dc040-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.058058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.058036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.058342 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.058323 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.064373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.064348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.064621 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.064572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xftm\" (UniqueName: \"kubernetes.io/projected/50fb299f-3f49-4a3d-9631-99d70d8dc040-kube-api-access-6xftm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-frw6q\" (UID: \"50fb299f-3f49-4a3d-9631-99d70d8dc040\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.177142 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.177108 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:23.352024 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.351998 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q"] Jul 09 19:47:23.353799 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:47:23.353773 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fb299f_3f49_4a3d_9631_99d70d8dc040.slice/crio-eb718d729bf4bd3a1385330bdd00fd9a9966b2742e24b02154cc36bd768f3849 WatchSource:0}: Error finding container eb718d729bf4bd3a1385330bdd00fd9a9966b2742e24b02154cc36bd768f3849: Status 404 returned error can't find the container with id eb718d729bf4bd3a1385330bdd00fd9a9966b2742e24b02154cc36bd768f3849 Jul 09 19:47:23.356044 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.356011 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:47:23.356126 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.356077 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:47:23.356126 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:23.356109 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:47:24.162934 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:24.162895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" event={"ID":"50fb299f-3f49-4a3d-9631-99d70d8dc040","Type":"ContainerStarted","Data":"3fc87e7c286a5f2ecd701caf7baad36b680039ca113c5f26452351a91a537864"} Jul 09 19:47:24.163418 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:24.162962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" event={"ID":"50fb299f-3f49-4a3d-9631-99d70d8dc040","Type":"ContainerStarted","Data":"eb718d729bf4bd3a1385330bdd00fd9a9966b2742e24b02154cc36bd768f3849"} Jul 09 19:47:24.177331 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:24.177301 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:24.181990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:24.181968 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:24.195708 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:24.195662 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" podStartSLOduration=2.195648085 podStartE2EDuration="2.195648085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:24.193284228 +0000 UTC m=+470.784044432" watchObservedRunningTime="2026-07-09 19:47:24.195648085 +0000 UTC m=+470.786408300" Jul 09 19:47:25.169058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:25.169026 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:25.170000 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:25.169982 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:47:27.625389 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:27.625342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:32.729251 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:32.729211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:32.830281 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:32.830246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:41.556906 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.556868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:41.562618 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.562575 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:41.582817 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.582776 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:41.582995 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.582884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.586030 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.586005 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 19:47:41.656217 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.656182 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:41.665496 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.665465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:41.721570 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.721532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.721769 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.721667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqswg\" (UniqueName: \"kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.724801 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.724775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:41.822383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.822281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqswg\" (UniqueName: \"kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.822383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.822348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.823092 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.823072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.832129 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.832100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqswg\" (UniqueName: \"kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg\") pod \"limitador-limitador-64c94fd64f-fsnn4\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:41.893767 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:41.893735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:42.053174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.053140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:42.055124 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:47:42.055098 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3521cc05_f9df_4591_94ac_8f4356eddc73.slice/crio-98204db44999698b89f211102f82717a3ce09e59675eef29e1287a944fe64f52 WatchSource:0}: Error finding container 98204db44999698b89f211102f82717a3ce09e59675eef29e1287a944fe64f52: Status 404 returned error can't find the container with id 98204db44999698b89f211102f82717a3ce09e59675eef29e1287a944fe64f52 Jul 09 19:47:42.265755 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.265724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" event={"ID":"3521cc05-f9df-4591-94ac-8f4356eddc73","Type":"ContainerStarted","Data":"98204db44999698b89f211102f82717a3ce09e59675eef29e1287a944fe64f52"} Jul 09 19:47:42.407662 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.407628 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:42.410806 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.410789 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:42.415404 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.415381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9w5tp\"" Jul 09 19:47:42.433337 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.433311 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:42.491627 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.491570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:42.531008 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.530924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscmp\" (UniqueName: \"kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp\") pod \"authorino-58cd955675-8ml8m\" (UID: \"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff\") " pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:42.590651 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.590619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:42.631361 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.631329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscmp\" (UniqueName: \"kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp\") pod \"authorino-58cd955675-8ml8m\" (UID: \"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff\") " pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:42.647053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.647023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscmp\" (UniqueName: \"kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp\") pod \"authorino-58cd955675-8ml8m\" (UID: \"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff\") " pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:42.720151 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.720089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:42.868334 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:42.867522 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:42.870709 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:47:42.870678 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23bd1823_cbfe_4fc5_91e3_a7d67c9984ff.slice/crio-10ba16e0075e3af07cc8e37b9fefcf34e47756e1341a2ac15238e16ac1a94a78 WatchSource:0}: Error finding container 10ba16e0075e3af07cc8e37b9fefcf34e47756e1341a2ac15238e16ac1a94a78: Status 404 returned error can't find the container with id 10ba16e0075e3af07cc8e37b9fefcf34e47756e1341a2ac15238e16ac1a94a78 Jul 09 19:47:43.273974 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:43.273937 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8ml8m" event={"ID":"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff","Type":"ContainerStarted","Data":"10ba16e0075e3af07cc8e37b9fefcf34e47756e1341a2ac15238e16ac1a94a78"} Jul 09 19:47:46.416248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:46.416209 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:47.301725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.301672 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8ml8m" event={"ID":"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff","Type":"ContainerStarted","Data":"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298"} Jul 09 19:47:47.301725 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.301702 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8ml8m" podUID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" containerName="authorino" containerID="cri-o://30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298" gracePeriod=30 Jul 09 19:47:47.303231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.303207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" event={"ID":"3521cc05-f9df-4591-94ac-8f4356eddc73","Type":"ContainerStarted","Data":"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d"} Jul 09 19:47:47.303354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.303264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:47.319724 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.319684 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8ml8m" podStartSLOduration=1.361711504 podStartE2EDuration="5.319671564s" totalImagesPullingTime="3.95796006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:47.317449892 +0000 UTC m=+493.908210098" watchObservedRunningTime="2026-07-09 19:47:47.319671564 +0000 UTC m=+493.910431768" Jul 09 19:47:47.556061 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.556003 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:47.575248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.575194 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" podStartSLOduration=1.809009037 podStartE2EDuration="6.575173002s" totalImagesPullingTime="4.766163965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:47:47.338155961 +0000 UTC m=+493.928916176" watchObservedRunningTime="2026-07-09 19:47:47.575173002 +0000 UTC m=+494.165933208" Jul 09 19:47:47.679783 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.679747 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kscmp\" (UniqueName: \"kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp\") pod \"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff\" (UID: \"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff\") " Jul 09 19:47:47.682258 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.682224 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp" (OuterVolumeSpecName: "kube-api-access-kscmp") pod "23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" (UID: "23bd1823-cbfe-4fc5-91e3-a7d67c9984ff"). InnerVolumeSpecName "kube-api-access-kscmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:47:47.780499 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:47.780462 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscmp\" (UniqueName: \"kubernetes.io/projected/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff-kube-api-access-kscmp\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:48.310064 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.310032 2562 generic.go:356] "Generic (PLEG): container finished" podID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" containerID="30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298" exitCode=0 Jul 09 19:47:48.310248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.310080 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8ml8m" Jul 09 19:47:48.310248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.310119 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8ml8m" event={"ID":"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff","Type":"ContainerDied","Data":"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298"} Jul 09 19:47:48.310248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.310171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8ml8m" event={"ID":"23bd1823-cbfe-4fc5-91e3-a7d67c9984ff","Type":"ContainerDied","Data":"10ba16e0075e3af07cc8e37b9fefcf34e47756e1341a2ac15238e16ac1a94a78"} Jul 09 19:47:48.310248 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.310202 2562 scope.go:117] "RemoveContainer" containerID="30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298" Jul 09 19:47:48.318896 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.318873 2562 scope.go:117] "RemoveContainer" containerID="30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298" Jul 09 19:47:48.319152 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:48.319137 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298\": container with ID starting with 30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298 not found: ID does not exist" containerID="30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298" Jul 09 19:47:48.319195 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.319160 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298"} err="failed to get container status \"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298\": rpc error: code = NotFound desc = could not find container \"30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298\": container with ID starting with 30048e73b88fd9dfb91b9affa68daa45aa2db03d467009691a8e73b8e2f52298 not found: ID does not exist" Jul 09 19:47:48.327633 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.327606 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:48.331870 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:48.331849 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8ml8m"] Jul 09 19:47:49.958221 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:49.958185 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" path="/var/lib/kubelet/pods/23bd1823-cbfe-4fc5-91e3-a7d67c9984ff/volumes" Jul 09 19:47:52.571679 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:52.571646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:52.634974 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:52.634946 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:47:56.408027 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:56.407995 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:56.408852 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:56.408308 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" podUID="3521cc05-f9df-4591-94ac-8f4356eddc73" containerName="limitador" containerID="cri-o://3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d" gracePeriod=30 Jul 09 19:47:56.409179 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:56.408907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:56.963038 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:56.963009 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:57.061088 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.060995 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file\") pod \"3521cc05-f9df-4591-94ac-8f4356eddc73\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " Jul 09 19:47:57.061292 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.061100 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqswg\" (UniqueName: \"kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg\") pod \"3521cc05-f9df-4591-94ac-8f4356eddc73\" (UID: \"3521cc05-f9df-4591-94ac-8f4356eddc73\") " Jul 09 19:47:57.061415 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.061390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file" (OuterVolumeSpecName: "config-file") pod "3521cc05-f9df-4591-94ac-8f4356eddc73" (UID: "3521cc05-f9df-4591-94ac-8f4356eddc73"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:47:57.063413 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.063384 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg" (OuterVolumeSpecName: "kube-api-access-nqswg") pod "3521cc05-f9df-4591-94ac-8f4356eddc73" (UID: "3521cc05-f9df-4591-94ac-8f4356eddc73"). InnerVolumeSpecName "kube-api-access-nqswg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:47:57.162104 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.162066 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqswg\" (UniqueName: \"kubernetes.io/projected/3521cc05-f9df-4591-94ac-8f4356eddc73-kube-api-access-nqswg\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:57.162104 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.162097 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3521cc05-f9df-4591-94ac-8f4356eddc73-config-file\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:47:57.360102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.360013 2562 generic.go:356] "Generic (PLEG): container finished" podID="3521cc05-f9df-4591-94ac-8f4356eddc73" containerID="3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d" exitCode=0 Jul 09 19:47:57.360102 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.360053 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" Jul 09 19:47:57.360291 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.360102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" event={"ID":"3521cc05-f9df-4591-94ac-8f4356eddc73","Type":"ContainerDied","Data":"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d"} Jul 09 19:47:57.360291 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.360158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fsnn4" event={"ID":"3521cc05-f9df-4591-94ac-8f4356eddc73","Type":"ContainerDied","Data":"98204db44999698b89f211102f82717a3ce09e59675eef29e1287a944fe64f52"} Jul 09 19:47:57.360291 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.360184 2562 scope.go:117] "RemoveContainer" containerID="3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d" Jul 09 19:47:57.369352 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.369334 2562 scope.go:117] "RemoveContainer" containerID="3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d" Jul 09 19:47:57.369653 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:47:57.369634 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d\": container with ID starting with 3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d not found: ID does not exist" containerID="3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d" Jul 09 19:47:57.369704 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.369663 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d"} err="failed to get container status \"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d\": rpc error: code = NotFound desc = could not find container \"3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d\": container with ID starting with 3ca5a1fb1866909ebb70d2219d3a97355afe14c8cddebaed221b3359c618a93d not found: ID does not exist" Jul 09 19:47:57.382501 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.382471 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:57.384486 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.384468 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fsnn4"] Jul 09 19:47:57.953356 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:47:57.953324 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3521cc05-f9df-4591-94ac-8f4356eddc73" path="/var/lib/kubelet/pods/3521cc05-f9df-4591-94ac-8f4356eddc73/volumes" Jul 09 19:48:01.875847 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:01.875811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:01.982449 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:01.982414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:02.317158 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317117 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sg56x"] Jul 09 19:48:02.317481 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317469 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3521cc05-f9df-4591-94ac-8f4356eddc73" containerName="limitador" Jul 09 19:48:02.317524 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317482 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3521cc05-f9df-4591-94ac-8f4356eddc73" containerName="limitador" Jul 09 19:48:02.317524 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317492 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" containerName="authorino" Jul 09 19:48:02.317524 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317497 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" containerName="authorino" Jul 09 19:48:02.317645 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317549 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bd1823-cbfe-4fc5-91e3-a7d67c9984ff" containerName="authorino" Jul 09 19:48:02.317645 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.317559 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3521cc05-f9df-4591-94ac-8f4356eddc73" containerName="limitador" Jul 09 19:48:02.321385 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.321366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.326152 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.326132 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zpvhr\"" Jul 09 19:48:02.326397 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.326377 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:48:02.327409 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.327392 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:48:02.327409 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.327398 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 19:48:02.337118 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.337092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sg56x"] Jul 09 19:48:02.368848 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.368812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:02.510673 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.510643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/680979e4-101f-4adb-a529-84424a175b22-data\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.510861 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.510737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc6m\" (UniqueName: \"kubernetes.io/projected/680979e4-101f-4adb-a529-84424a175b22-kube-api-access-6xc6m\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.611549 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.611457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc6m\" (UniqueName: \"kubernetes.io/projected/680979e4-101f-4adb-a529-84424a175b22-kube-api-access-6xc6m\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.611549 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.611506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/680979e4-101f-4adb-a529-84424a175b22-data\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.611853 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.611838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/680979e4-101f-4adb-a529-84424a175b22-data\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.622119 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.622093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc6m\" (UniqueName: \"kubernetes.io/projected/680979e4-101f-4adb-a529-84424a175b22-kube-api-access-6xc6m\") pod \"postgres-69fb4594cf-sg56x\" (UID: \"680979e4-101f-4adb-a529-84424a175b22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.631918 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.631893 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:02.774530 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:02.774499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sg56x"] Jul 09 19:48:02.775842 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:02.775815 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680979e4_101f_4adb_a529_84424a175b22.slice/crio-9730d8a1c0dc26fbf6b02753cc0bc403edcb8e6fd1ae88a9599add4b7df98cb7 WatchSource:0}: Error finding container 9730d8a1c0dc26fbf6b02753cc0bc403edcb8e6fd1ae88a9599add4b7df98cb7: Status 404 returned error can't find the container with id 9730d8a1c0dc26fbf6b02753cc0bc403edcb8e6fd1ae88a9599add4b7df98cb7 Jul 09 19:48:03.392654 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:03.392621 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" event={"ID":"680979e4-101f-4adb-a529-84424a175b22","Type":"ContainerStarted","Data":"9730d8a1c0dc26fbf6b02753cc0bc403edcb8e6fd1ae88a9599add4b7df98cb7"} Jul 09 19:48:08.427528 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:08.427484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" event={"ID":"680979e4-101f-4adb-a529-84424a175b22","Type":"ContainerStarted","Data":"fb95a5bed70b150d695495cd459d48d9b95feafdb03418e1cc59e7af35ec25c4"} Jul 09 19:48:08.428100 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:08.427626 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:08.443475 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:08.443423 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" podStartSLOduration=1.5710228960000001 podStartE2EDuration="6.443410468s" totalImagesPullingTime="4.872387572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:48:08.442354155 +0000 UTC m=+515.033114381" watchObservedRunningTime="2026-07-09 19:48:08.443410468 +0000 UTC m=+515.034170673" Jul 09 19:48:14.463392 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:14.463361 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sg56x" Jul 09 19:48:14.791919 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:14.791826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:14.945240 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:14.945204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:15.034108 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.034069 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:15.040264 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.040236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:15.043309 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.043241 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9w5tp\"" Jul 09 19:48:15.046309 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.046287 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:15.103855 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.103820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vj8\" (UniqueName: \"kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8\") pod \"authorino-5d4b67c446-6d984\" (UID: \"cf02b389-b5ac-40f7-b056-4814c554ff08\") " pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:15.204940 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.204902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vj8\" (UniqueName: \"kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8\") pod \"authorino-5d4b67c446-6d984\" (UID: \"cf02b389-b5ac-40f7-b056-4814c554ff08\") " pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:15.214231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.214203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vj8\" (UniqueName: \"kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8\") pod \"authorino-5d4b67c446-6d984\" (UID: \"cf02b389-b5ac-40f7-b056-4814c554ff08\") " pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:15.289397 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.289366 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:15.289660 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.289646 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:15.317618 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.317594 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:48:15.323111 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.323084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.325968 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.325951 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 19:48:15.332328 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.332301 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:48:15.406452 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.406410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.406702 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.406459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fsh\" (UniqueName: \"kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.431732 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.431709 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:15.433653 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:15.433626 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf02b389_b5ac_40f7_b056_4814c554ff08.slice/crio-a55f6d35474725d067adb3aae65746e39dee206ade5def5d8e0103b2fb254a10 WatchSource:0}: Error finding container a55f6d35474725d067adb3aae65746e39dee206ade5def5d8e0103b2fb254a10: Status 404 returned error can't find the container with id a55f6d35474725d067adb3aae65746e39dee206ade5def5d8e0103b2fb254a10 Jul 09 19:48:15.464345 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.464311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6d984" event={"ID":"cf02b389-b5ac-40f7-b056-4814c554ff08","Type":"ContainerStarted","Data":"a55f6d35474725d067adb3aae65746e39dee206ade5def5d8e0103b2fb254a10"} Jul 09 19:48:15.507808 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.507775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.507944 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.507817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fsh\" (UniqueName: \"kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.510389 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.510367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.522544 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.522522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fsh\" (UniqueName: \"kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh\") pod \"authorino-686944fc45-jbxr9\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.636351 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.636286 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:48:15.773805 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:15.773782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:48:15.777454 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:15.777429 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec928d9_81c2_4454_80dd_ccd650e2fcc2.slice/crio-47339382eb98489198eeae43525b392b76e19165d14b4c079731d5f962946762 WatchSource:0}: Error finding container 47339382eb98489198eeae43525b392b76e19165d14b4c079731d5f962946762: Status 404 returned error can't find the container with id 47339382eb98489198eeae43525b392b76e19165d14b4c079731d5f962946762 Jul 09 19:48:16.471342 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.471301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6d984" event={"ID":"cf02b389-b5ac-40f7-b056-4814c554ff08","Type":"ContainerStarted","Data":"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62"} Jul 09 19:48:16.471844 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.471359 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-6d984" podUID="cf02b389-b5ac-40f7-b056-4814c554ff08" containerName="authorino" containerID="cri-o://647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62" gracePeriod=30 Jul 09 19:48:16.472897 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.472868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686944fc45-jbxr9" event={"ID":"aec928d9-81c2-4454-80dd-ccd650e2fcc2","Type":"ContainerStarted","Data":"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620"} Jul 09 19:48:16.472981 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.472908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686944fc45-jbxr9" event={"ID":"aec928d9-81c2-4454-80dd-ccd650e2fcc2","Type":"ContainerStarted","Data":"47339382eb98489198eeae43525b392b76e19165d14b4c079731d5f962946762"} Jul 09 19:48:16.492781 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.492059 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-6d984" podStartSLOduration=1.146759585 podStartE2EDuration="1.492041591s" totalImagesPullingTime="345.282006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:48:16.490604049 +0000 UTC m=+523.081364245" watchObservedRunningTime="2026-07-09 19:48:16.492041591 +0000 UTC m=+523.082801797" Jul 09 19:48:16.508731 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.508693 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-686944fc45-jbxr9" podStartSLOduration=1.181578206 podStartE2EDuration="1.508680611s" totalImagesPullingTime="327.102405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:48:16.507446991 +0000 UTC m=+523.098207198" watchObservedRunningTime="2026-07-09 19:48:16.508680611 +0000 UTC m=+523.099440812" Jul 09 19:48:16.719759 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.719738 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:16.818031 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.817953 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6vj8\" (UniqueName: \"kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8\") pod \"cf02b389-b5ac-40f7-b056-4814c554ff08\" (UID: \"cf02b389-b5ac-40f7-b056-4814c554ff08\") " Jul 09 19:48:16.820088 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.820060 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8" (OuterVolumeSpecName: "kube-api-access-l6vj8") pod "cf02b389-b5ac-40f7-b056-4814c554ff08" (UID: "cf02b389-b5ac-40f7-b056-4814c554ff08"). InnerVolumeSpecName "kube-api-access-l6vj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:48:16.919311 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:16.919274 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vj8\" (UniqueName: \"kubernetes.io/projected/cf02b389-b5ac-40f7-b056-4814c554ff08-kube-api-access-l6vj8\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:48:17.479407 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.479373 2562 generic.go:356] "Generic (PLEG): container finished" podID="cf02b389-b5ac-40f7-b056-4814c554ff08" containerID="647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62" exitCode=0 Jul 09 19:48:17.479810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.479419 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6d984" Jul 09 19:48:17.479810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.479470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6d984" event={"ID":"cf02b389-b5ac-40f7-b056-4814c554ff08","Type":"ContainerDied","Data":"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62"} Jul 09 19:48:17.479810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.479511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6d984" event={"ID":"cf02b389-b5ac-40f7-b056-4814c554ff08","Type":"ContainerDied","Data":"a55f6d35474725d067adb3aae65746e39dee206ade5def5d8e0103b2fb254a10"} Jul 09 19:48:17.479810 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.479533 2562 scope.go:117] "RemoveContainer" containerID="647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62" Jul 09 19:48:17.489148 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.489123 2562 scope.go:117] "RemoveContainer" containerID="647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62" Jul 09 19:48:17.489427 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:48:17.489408 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62\": container with ID starting with 647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62 not found: ID does not exist" containerID="647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62" Jul 09 19:48:17.489499 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.489438 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62"} err="failed to get container status \"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62\": rpc error: code = NotFound desc = could not find container \"647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62\": container with ID starting with 647beab655e6e17b7a965bde1584d1de8b248fa2d08f4b4b16fb288344e68d62 not found: ID does not exist" Jul 09 19:48:17.507935 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.507907 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:17.513205 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.513184 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6d984"] Jul 09 19:48:17.953075 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:17.953043 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf02b389-b5ac-40f7-b056-4814c554ff08" path="/var/lib/kubelet/pods/cf02b389-b5ac-40f7-b056-4814c554ff08/volumes" Jul 09 19:48:20.999627 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:20.999593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:21.021736 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.021706 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:21.022128 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.022112 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf02b389-b5ac-40f7-b056-4814c554ff08" containerName="authorino" Jul 09 19:48:21.022178 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.022130 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf02b389-b5ac-40f7-b056-4814c554ff08" containerName="authorino" Jul 09 19:48:21.022213 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.022193 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf02b389-b5ac-40f7-b056-4814c554ff08" containerName="authorino" Jul 09 19:48:21.084424 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.084391 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:21.084621 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.084493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.087594 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.087551 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 19:48:21.087594 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.087566 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r8qmr\"" Jul 09 19:48:21.087832 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.087606 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 19:48:21.098688 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.098667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:21.153950 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.153917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.154111 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.153963 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sscx\" (UniqueName: \"kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.198218 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.198158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:21.199607 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.199570 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:48:21.224658 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.224630 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:48:21.224805 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.224699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.255367 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.255297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjz8\" (UniqueName: \"kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.255367 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.255341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.255541 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.255413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sscx\" (UniqueName: \"kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.255541 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.255464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.257800 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.257782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.264680 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.264656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sscx\" (UniqueName: \"kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx\") pod \"maas-controller-c77549cfc-8fg57\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.266310 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.266275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:21.356844 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.356809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.357031 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.356859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjz8\" (UniqueName: \"kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.359443 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.359423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.366458 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.366433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjz8\" (UniqueName: \"kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8\") pod \"maas-controller-6885c5f9d9-45kfh\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.394619 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.394565 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:21.535535 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.535464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:21.555638 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.555612 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:21.557151 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:21.557115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224baabf_53ed_47cb_948f_99028b1c1050.slice/crio-bd3ddb2080e5fb9c2a602944e6706049ea634bb485bad999e23589bcbe4cdbb4 WatchSource:0}: Error finding container bd3ddb2080e5fb9c2a602944e6706049ea634bb485bad999e23589bcbe4cdbb4: Status 404 returned error can't find the container with id bd3ddb2080e5fb9c2a602944e6706049ea634bb485bad999e23589bcbe4cdbb4 Jul 09 19:48:21.703516 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:21.703491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:48:21.704674 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:21.704648 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32ed21_3e2e_4b56_b9b2_2c954033f50d.slice/crio-27cbb9c75d48273d7e6b59cec1ea1326a65230f2fdf010ac5c30f5b3d83948bc WatchSource:0}: Error finding container 27cbb9c75d48273d7e6b59cec1ea1326a65230f2fdf010ac5c30f5b3d83948bc: Status 404 returned error can't find the container with id 27cbb9c75d48273d7e6b59cec1ea1326a65230f2fdf010ac5c30f5b3d83948bc Jul 09 19:48:22.513447 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:22.513407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c77549cfc-8fg57" event={"ID":"224baabf-53ed-47cb-948f-99028b1c1050","Type":"ContainerStarted","Data":"bd3ddb2080e5fb9c2a602944e6706049ea634bb485bad999e23589bcbe4cdbb4"} Jul 09 19:48:22.515244 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:22.515182 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" event={"ID":"ba32ed21-3e2e-4b56-b9b2-2c954033f50d","Type":"ContainerStarted","Data":"27cbb9c75d48273d7e6b59cec1ea1326a65230f2fdf010ac5c30f5b3d83948bc"} Jul 09 19:48:23.902785 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:23.902703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:24.002175 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.002111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:24.530058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.530018 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c77549cfc-8fg57" event={"ID":"224baabf-53ed-47cb-948f-99028b1c1050","Type":"ContainerStarted","Data":"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934"} Jul 09 19:48:24.530247 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.530149 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:24.531624 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.531599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" event={"ID":"ba32ed21-3e2e-4b56-b9b2-2c954033f50d","Type":"ContainerStarted","Data":"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb"} Jul 09 19:48:24.531761 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.531656 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:24.553251 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.553195 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c77549cfc-8fg57" podStartSLOduration=2.45574929 podStartE2EDuration="4.553182971s" totalImagesPullingTime="2.097433681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:48:24.551765477 +0000 UTC m=+531.142525694" watchObservedRunningTime="2026-07-09 19:48:24.553182971 +0000 UTC m=+531.143943176" Jul 09 19:48:24.573202 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:24.573116 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" podStartSLOduration=1.633598696 podStartE2EDuration="3.573074975s" totalImagesPullingTime="1.939476279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:48:24.571612614 +0000 UTC m=+531.162372825" watchObservedRunningTime="2026-07-09 19:48:24.573074975 +0000 UTC m=+531.163835224" Jul 09 19:48:32.933062 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:32.933016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:35.538618 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.538566 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:48:35.538984 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.538779 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:35.594833 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.594799 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:35.596039 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.595978 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c77549cfc-8fg57" podUID="224baabf-53ed-47cb-948f-99028b1c1050" containerName="manager" containerID="cri-o://a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934" gracePeriod=30 Jul 09 19:48:35.839966 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.839944 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:35.879841 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.879811 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert\") pod \"224baabf-53ed-47cb-948f-99028b1c1050\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " Jul 09 19:48:35.879985 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.879899 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sscx\" (UniqueName: \"kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx\") pod \"224baabf-53ed-47cb-948f-99028b1c1050\" (UID: \"224baabf-53ed-47cb-948f-99028b1c1050\") " Jul 09 19:48:35.882231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.882202 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "224baabf-53ed-47cb-948f-99028b1c1050" (UID: "224baabf-53ed-47cb-948f-99028b1c1050"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:48:35.882231 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.882201 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx" (OuterVolumeSpecName: "kube-api-access-4sscx") pod "224baabf-53ed-47cb-948f-99028b1c1050" (UID: "224baabf-53ed-47cb-948f-99028b1c1050"). InnerVolumeSpecName "kube-api-access-4sscx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:48:35.980838 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.980811 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sscx\" (UniqueName: \"kubernetes.io/projected/224baabf-53ed-47cb-948f-99028b1c1050-kube-api-access-4sscx\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:48:35.980838 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:35.980837 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/224baabf-53ed-47cb-948f-99028b1c1050-webhook-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:48:36.603055 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.603025 2562 generic.go:356] "Generic (PLEG): container finished" podID="224baabf-53ed-47cb-948f-99028b1c1050" containerID="a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934" exitCode=0 Jul 09 19:48:36.603527 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.603069 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c77549cfc-8fg57" Jul 09 19:48:36.603527 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.603112 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c77549cfc-8fg57" event={"ID":"224baabf-53ed-47cb-948f-99028b1c1050","Type":"ContainerDied","Data":"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934"} Jul 09 19:48:36.603527 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.603181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c77549cfc-8fg57" event={"ID":"224baabf-53ed-47cb-948f-99028b1c1050","Type":"ContainerDied","Data":"bd3ddb2080e5fb9c2a602944e6706049ea634bb485bad999e23589bcbe4cdbb4"} Jul 09 19:48:36.603527 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.603216 2562 scope.go:117] "RemoveContainer" containerID="a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934" Jul 09 19:48:36.612403 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.612386 2562 scope.go:117] "RemoveContainer" containerID="a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934" Jul 09 19:48:36.612724 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:48:36.612705 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934\": container with ID starting with a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934 not found: ID does not exist" containerID="a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934" Jul 09 19:48:36.612802 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.612733 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934"} err="failed to get container status \"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934\": rpc error: code = NotFound desc = could not find container \"a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934\": container with ID starting with a1f989ae359d2c543cbf4721c3dd0591fe2a3d831333f5e2d68e73181f5c5934 not found: ID does not exist" Jul 09 19:48:36.623019 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.622995 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:36.631319 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:36.631296 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c77549cfc-8fg57"] Jul 09 19:48:37.953446 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:37.953403 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224baabf-53ed-47cb-948f-99028b1c1050" path="/var/lib/kubelet/pods/224baabf-53ed-47cb-948f-99028b1c1050/volumes" Jul 09 19:48:46.152634 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:46.152592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:56.325497 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.325465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:56.422141 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.422107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:56.788059 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.788011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:56.912009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.911975 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g"] Jul 09 19:48:56.912377 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.912364 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224baabf-53ed-47cb-948f-99028b1c1050" containerName="manager" Jul 09 19:48:56.912425 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.912379 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="224baabf-53ed-47cb-948f-99028b1c1050" containerName="manager" Jul 09 19:48:56.912471 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.912441 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="224baabf-53ed-47cb-948f-99028b1c1050" containerName="manager" Jul 09 19:48:56.915554 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.915533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:56.918341 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.918322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 19:48:56.918455 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.918440 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d9hhm\"" Jul 09 19:48:56.927543 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:56.927520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g"] Jul 09 19:48:57.057985 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.057900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.058132 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.058043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cedb07c-f373-4c45-93b2-13a10fedbc2e-openshift-service-ca\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.058132 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.058122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzd9t\" (UniqueName: \"kubernetes.io/projected/9cedb07c-f373-4c45-93b2-13a10fedbc2e-kube-api-access-wzd9t\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.159332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.159297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.159502 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.159361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cedb07c-f373-4c45-93b2-13a10fedbc2e-openshift-service-ca\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.159502 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.159405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzd9t\" (UniqueName: \"kubernetes.io/projected/9cedb07c-f373-4c45-93b2-13a10fedbc2e-kube-api-access-wzd9t\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.159502 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:48:57.159459 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 09 19:48:57.159716 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:48:57.159543 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls podName:9cedb07c-f373-4c45-93b2-13a10fedbc2e nodeName:}" failed. No retries permitted until 2026-07-09 19:48:57.659525522 +0000 UTC m=+564.250285707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls") pod "maas-api-7f7dd46c4-h6r9g" (UID: "9cedb07c-f373-4c45-93b2-13a10fedbc2e") : secret "maas-api-serving-cert" not found Jul 09 19:48:57.159950 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.159933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cedb07c-f373-4c45-93b2-13a10fedbc2e-openshift-service-ca\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.168827 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.168800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzd9t\" (UniqueName: \"kubernetes.io/projected/9cedb07c-f373-4c45-93b2-13a10fedbc2e-kube-api-access-wzd9t\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.351199 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.351110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:57.450707 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.450675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:57.545764 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.545729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:48:57.550395 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.550369 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lch5w"] Jul 09 19:48:57.553980 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.553959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.557539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.557521 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 19:48:57.567515 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.567489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lch5w"] Jul 09 19:48:57.583740 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.583716 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7ptzq"] Jul 09 19:48:57.587053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.587033 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.589569 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.589544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t9gkg\"" Jul 09 19:48:57.597960 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.597938 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7ptzq"] Jul 09 19:48:57.663870 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.663840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6a813a-2aae-4908-9561-5f9b79fc8857-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.664053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.663893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.664053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.663948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vqt\" (UniqueName: \"kubernetes.io/projected/3d6a813a-2aae-4908-9561-5f9b79fc8857-kube-api-access-x8vqt\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.664053 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.663977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/2ee1edf3-ea29-47fb-be62-cc8260de47a1-kube-api-access-8rlbk\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.664195 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.664117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1edf3-ea29-47fb-be62-cc8260de47a1-plugins-config-volume\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.666608 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.666554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cedb07c-f373-4c45-93b2-13a10fedbc2e-maas-api-tls\") pod \"maas-api-7f7dd46c4-h6r9g\" (UID: \"9cedb07c-f373-4c45-93b2-13a10fedbc2e\") " pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.765132 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1edf3-ea29-47fb-be62-cc8260de47a1-plugins-config-volume\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.765354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6a813a-2aae-4908-9561-5f9b79fc8857-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.765354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vqt\" (UniqueName: \"kubernetes.io/projected/3d6a813a-2aae-4908-9561-5f9b79fc8857-kube-api-access-x8vqt\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.765354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/2ee1edf3-ea29-47fb-be62-cc8260de47a1-kube-api-access-8rlbk\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.765919 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765898 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d6a813a-2aae-4908-9561-5f9b79fc8857-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.765993 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.765975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1edf3-ea29-47fb-be62-cc8260de47a1-plugins-config-volume\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.773887 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.773861 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/2ee1edf3-ea29-47fb-be62-cc8260de47a1-kube-api-access-8rlbk\") pod \"payload-processing-58f86c7798-7ptzq\" (UID: \"2ee1edf3-ea29-47fb-be62-cc8260de47a1\") " pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:57.774031 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.773863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vqt\" (UniqueName: \"kubernetes.io/projected/3d6a813a-2aae-4908-9561-5f9b79fc8857-kube-api-access-x8vqt\") pod \"payload-pre-processing-7964457696-lch5w\" (UID: \"3d6a813a-2aae-4908-9561-5f9b79fc8857\") " pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.826056 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.826017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:48:57.864983 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.864948 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:48:57.897696 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:57.897643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:48:58.014893 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.014858 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g"] Jul 09 19:48:58.016961 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:58.016930 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cedb07c_f373_4c45_93b2_13a10fedbc2e.slice/crio-40362b218274f44b53a234cb05985cdfa8dc440440223f90411c06bdf30167a0 WatchSource:0}: Error finding container 40362b218274f44b53a234cb05985cdfa8dc440440223f90411c06bdf30167a0: Status 404 returned error can't find the container with id 40362b218274f44b53a234cb05985cdfa8dc440440223f90411c06bdf30167a0 Jul 09 19:48:58.067926 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.067899 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lch5w"] Jul 09 19:48:58.069963 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:58.069932 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6a813a_2aae_4908_9561_5f9b79fc8857.slice/crio-08b5b94ad68f639f57e3bf444dbf220282ba24b71c1e213b68c40a6c93c6b613 WatchSource:0}: Error finding container 08b5b94ad68f639f57e3bf444dbf220282ba24b71c1e213b68c40a6c93c6b613: Status 404 returned error can't find the container with id 08b5b94ad68f639f57e3bf444dbf220282ba24b71c1e213b68c40a6c93c6b613 Jul 09 19:48:58.092855 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.092831 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7ptzq"] Jul 09 19:48:58.093339 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:48:58.093315 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee1edf3_ea29_47fb_be62_cc8260de47a1.slice/crio-a72cb7a13e1a2f7c400c2d788e00119fe6bcb5d7f83c6d27be6fa5d10fc638a7 WatchSource:0}: Error finding container a72cb7a13e1a2f7c400c2d788e00119fe6bcb5d7f83c6d27be6fa5d10fc638a7: Status 404 returned error can't find the container with id a72cb7a13e1a2f7c400c2d788e00119fe6bcb5d7f83c6d27be6fa5d10fc638a7 Jul 09 19:48:58.729496 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.729390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" event={"ID":"3d6a813a-2aae-4908-9561-5f9b79fc8857","Type":"ContainerStarted","Data":"08b5b94ad68f639f57e3bf444dbf220282ba24b71c1e213b68c40a6c93c6b613"} Jul 09 19:48:58.731876 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.731790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" event={"ID":"2ee1edf3-ea29-47fb-be62-cc8260de47a1","Type":"ContainerStarted","Data":"a72cb7a13e1a2f7c400c2d788e00119fe6bcb5d7f83c6d27be6fa5d10fc638a7"} Jul 09 19:48:58.733941 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:48:58.733865 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" event={"ID":"9cedb07c-f373-4c45-93b2-13a10fedbc2e","Type":"ContainerStarted","Data":"40362b218274f44b53a234cb05985cdfa8dc440440223f90411c06bdf30167a0"} Jul 09 19:49:00.766984 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.766887 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" event={"ID":"3d6a813a-2aae-4908-9561-5f9b79fc8857","Type":"ContainerStarted","Data":"8c4fa1534b96f4dc96f63d2a2d304a1743a4cbbc05da8c7f878885a571c66b7b"} Jul 09 19:49:00.767530 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.767491 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:49:00.768790 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.768768 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" event={"ID":"2ee1edf3-ea29-47fb-be62-cc8260de47a1","Type":"ContainerStarted","Data":"a1ce8457e99a3e8e4369e9f636b6d3361b0bb23689be1546222d9133d0432a19"} Jul 09 19:49:00.768937 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.768881 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:49:00.770930 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.770903 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" event={"ID":"9cedb07c-f373-4c45-93b2-13a10fedbc2e","Type":"ContainerStarted","Data":"9b98d0be16885d051cd5d37cfdfd0751d68fe0e26209c5a49d040fa98e85fb33"} Jul 09 19:49:00.771174 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.771157 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:49:00.785814 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.785776 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" podStartSLOduration=1.440251196 podStartE2EDuration="3.785763123s" totalImagesPullingTime="2.345511927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:49:00.785186536 +0000 UTC m=+567.375946770" watchObservedRunningTime="2026-07-09 19:49:00.785763123 +0000 UTC m=+567.376523318" Jul 09 19:49:00.806090 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.806046 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" podStartSLOduration=2.409814908 podStartE2EDuration="4.806033994s" totalImagesPullingTime="2.396219086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:49:00.803521976 +0000 UTC m=+567.394282181" watchObservedRunningTime="2026-07-09 19:49:00.806033994 +0000 UTC m=+567.396794200" Jul 09 19:49:00.821643 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:00.821602 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" podStartSLOduration=1.5020134170000001 podStartE2EDuration="3.821562224s" totalImagesPullingTime="2.319548807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:49:00.820357907 +0000 UTC m=+567.411118121" watchObservedRunningTime="2026-07-09 19:49:00.821562224 +0000 UTC m=+567.412322430" Jul 09 19:49:05.799872 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.799785 2562 generic.go:356] "Generic (PLEG): container finished" podID="50fb299f-3f49-4a3d-9631-99d70d8dc040" containerID="3fc87e7c286a5f2ecd701caf7baad36b680039ca113c5f26452351a91a537864" exitCode=0 Jul 09 19:49:05.800293 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.799863 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" event={"ID":"50fb299f-3f49-4a3d-9631-99d70d8dc040","Type":"ContainerDied","Data":"3fc87e7c286a5f2ecd701caf7baad36b680039ca113c5f26452351a91a537864"} Jul 09 19:49:05.800346 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.800294 2562 scope.go:117] "RemoveContainer" containerID="3fc87e7c286a5f2ecd701caf7baad36b680039ca113c5f26452351a91a537864" Jul 09 19:49:05.801289 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.801261 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:49:05.801389 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.801345 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:49:05.801452 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:05.801392 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 19:49:06.782214 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:06.782187 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f7dd46c4-h6r9g" Jul 09 19:49:06.807525 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:06.807489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" event={"ID":"50fb299f-3f49-4a3d-9631-99d70d8dc040","Type":"ContainerStarted","Data":"85e24695ce9890f08efaf2b7c34d9b0c381eedb5ea269af4cf49a08e027c9ade"} Jul 09 19:49:07.178000 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:07.177960 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:49:07.179215 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:07.179190 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-frw6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 09 19:49:07.179332 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:07.179239 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" podUID="50fb299f-3f49-4a3d-9631-99d70d8dc040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 09 19:49:07.783609 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:07.783543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:07.881243 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:07.881195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:08.177984 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:08.177951 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-frw6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 09 19:49:08.178179 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:08.178013 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" podUID="50fb299f-3f49-4a3d-9631-99d70d8dc040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 09 19:49:09.177955 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:09.177916 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-frw6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 09 19:49:09.178349 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:09.177977 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" podUID="50fb299f-3f49-4a3d-9631-99d70d8dc040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 09 19:49:10.169450 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:10.169413 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:49:10.178312 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:10.178273 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-frw6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 09 19:49:10.178694 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:10.178334 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" podUID="50fb299f-3f49-4a3d-9631-99d70d8dc040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 09 19:49:11.181342 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:11.181306 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:49:11.182492 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:11.182472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-frw6q" Jul 09 19:49:11.778873 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:11.778840 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-lch5w" Jul 09 19:49:11.779096 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:11.778931 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-7ptzq" Jul 09 19:49:22.038944 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:22.038896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:33.892411 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:33.892374 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:49:33.893720 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:33.893696 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:49:39.129439 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:39.129397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:45.509163 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:45.509132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:56.795270 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.795173 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg"] Jul 09 19:49:56.797923 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.797906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.802052 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.802028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 19:49:56.802224 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.802028 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 19:49:56.802224 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.802067 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 19:49:56.802354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.802341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mpnwx\"" Jul 09 19:49:56.811781 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.811759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg"] Jul 09 19:49:56.823046 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.823021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:49:56.897427 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.897427 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm8q\" (UniqueName: \"kubernetes.io/projected/71f2197e-0f37-410d-875c-2c46b4305f7b-kube-api-access-5fm8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.897653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.897653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.897653 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2197e-0f37-410d-875c-2c46b4305f7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.897759 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.897695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998197 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998375 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm8q\" (UniqueName: \"kubernetes.io/projected/71f2197e-0f37-410d-875c-2c46b4305f7b-kube-api-access-5fm8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998485 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998574 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998574 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2197e-0f37-410d-875c-2c46b4305f7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998727 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998843 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:56.998925 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:56.998887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:57.001007 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.000981 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2197e-0f37-410d-875c-2c46b4305f7b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:57.001319 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.001299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2197e-0f37-410d-875c-2c46b4305f7b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:57.006921 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.006902 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm8q\" (UniqueName: \"kubernetes.io/projected/71f2197e-0f37-410d-875c-2c46b4305f7b-kube-api-access-5fm8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg\" (UID: \"71f2197e-0f37-410d-875c-2c46b4305f7b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:57.107914 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.107836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:49:57.248898 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.248869 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg"] Jul 09 19:49:57.250676 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:49:57.250651 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f2197e_0f37_410d_875c_2c46b4305f7b.slice/crio-37ec697d56876ed8bc39e877ba0bde4ee9d2486b3304764091a48c9ffa3f1d8d WatchSource:0}: Error finding container 37ec697d56876ed8bc39e877ba0bde4ee9d2486b3304764091a48c9ffa3f1d8d: Status 404 returned error can't find the container with id 37ec697d56876ed8bc39e877ba0bde4ee9d2486b3304764091a48c9ffa3f1d8d Jul 09 19:49:57.252373 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:57.252355 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:49:58.102854 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:58.102809 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" event={"ID":"71f2197e-0f37-410d-875c-2c46b4305f7b","Type":"ContainerStarted","Data":"37ec697d56876ed8bc39e877ba0bde4ee9d2486b3304764091a48c9ffa3f1d8d"} Jul 09 19:49:59.111516 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:59.111474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" event={"ID":"71f2197e-0f37-410d-875c-2c46b4305f7b","Type":"ContainerStarted","Data":"9bfc06a7c7557446c8cd77d0c5f12796ab5748f24be50cc740c809decb8e2b70"} Jul 09 19:49:59.131903 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:49:59.131838 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" podStartSLOduration=1.951055432 podStartE2EDuration="3.13181805s" totalImagesPullingTime="1.180762618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:49:59.131568585 +0000 UTC m=+625.722328790" watchObservedRunningTime="2026-07-09 19:49:59.13181805 +0000 UTC m=+625.722578258" Jul 09 19:50:07.108658 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:07.108614 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:50:07.109058 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:07.108676 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:50:07.121009 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:07.120983 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:50:07.170211 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:07.170166 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg" Jul 09 19:50:07.228533 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:07.228502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:50:26.635673 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:26.635639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:50:31.852487 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:50:31.852449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:14.957320 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:14.957284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:42.932460 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:42.932360 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84445d6688-df7rg"] Jul 09 19:51:42.935327 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:42.935304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:42.946171 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:42.946148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84445d6688-df7rg"] Jul 09 19:51:43.042892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.042847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac93773-1f97-4770-a62b-a0d7f563fa1c-tls-cert\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.043085 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.042927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhh9\" (UniqueName: \"kubernetes.io/projected/7ac93773-1f97-4770-a62b-a0d7f563fa1c-kube-api-access-bdhh9\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.143809 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.143781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac93773-1f97-4770-a62b-a0d7f563fa1c-tls-cert\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.143965 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.143818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhh9\" (UniqueName: \"kubernetes.io/projected/7ac93773-1f97-4770-a62b-a0d7f563fa1c-kube-api-access-bdhh9\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.146322 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.146299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac93773-1f97-4770-a62b-a0d7f563fa1c-tls-cert\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.153121 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.153100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhh9\" (UniqueName: \"kubernetes.io/projected/7ac93773-1f97-4770-a62b-a0d7f563fa1c-kube-api-access-bdhh9\") pod \"authorino-84445d6688-df7rg\" (UID: \"7ac93773-1f97-4770-a62b-a0d7f563fa1c\") " pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.244382 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.244304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84445d6688-df7rg" Jul 09 19:51:43.279054 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.279017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:43.379485 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.379446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:43.385792 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.385770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84445d6688-df7rg"] Jul 09 19:51:43.387010 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:51:43.386988 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac93773_1f97_4770_a62b_a0d7f563fa1c.slice/crio-06cb18df26aa223ee0f319e57a4f946de5fdb38c66ae5f65af66490282d8cce3 WatchSource:0}: Error finding container 06cb18df26aa223ee0f319e57a4f946de5fdb38c66ae5f65af66490282d8cce3: Status 404 returned error can't find the container with id 06cb18df26aa223ee0f319e57a4f946de5fdb38c66ae5f65af66490282d8cce3 Jul 09 19:51:43.689644 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:43.689610 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84445d6688-df7rg" event={"ID":"7ac93773-1f97-4770-a62b-a0d7f563fa1c","Type":"ContainerStarted","Data":"06cb18df26aa223ee0f319e57a4f946de5fdb38c66ae5f65af66490282d8cce3"} Jul 09 19:51:44.465541 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.465497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:44.574728 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.574685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:51:44.699383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.699332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84445d6688-df7rg" event={"ID":"7ac93773-1f97-4770-a62b-a0d7f563fa1c","Type":"ContainerStarted","Data":"9fcef39125fb04eafc2a18d5b60904d41f28b2a15e2f8e4c1f09c2abc4802528"} Jul 09 19:51:44.723383 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.723276 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84445d6688-df7rg" podStartSLOduration=2.190568919 podStartE2EDuration="2.723261067s" totalImagesPullingTime="532.692148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:51:44.718980328 +0000 UTC m=+731.309740530" watchObservedRunningTime="2026-07-09 19:51:44.723261067 +0000 UTC m=+731.314021290" Jul 09 19:51:44.763139 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.763102 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:51:44.763386 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:44.763355 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-686944fc45-jbxr9" podUID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" containerName="authorino" containerID="cri-o://a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620" gracePeriod=30 Jul 09 19:51:45.022901 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.022876 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:51:45.062902 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.062874 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65fsh\" (UniqueName: \"kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh\") pod \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " Jul 09 19:51:45.063103 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.062918 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert\") pod \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\" (UID: \"aec928d9-81c2-4454-80dd-ccd650e2fcc2\") " Jul 09 19:51:45.065286 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.065252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh" (OuterVolumeSpecName: "kube-api-access-65fsh") pod "aec928d9-81c2-4454-80dd-ccd650e2fcc2" (UID: "aec928d9-81c2-4454-80dd-ccd650e2fcc2"). InnerVolumeSpecName "kube-api-access-65fsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:51:45.076962 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.076925 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "aec928d9-81c2-4454-80dd-ccd650e2fcc2" (UID: "aec928d9-81c2-4454-80dd-ccd650e2fcc2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:51:45.163861 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.163822 2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aec928d9-81c2-4454-80dd-ccd650e2fcc2-tls-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:51:45.163861 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.163863 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fsh\" (UniqueName: \"kubernetes.io/projected/aec928d9-81c2-4454-80dd-ccd650e2fcc2-kube-api-access-65fsh\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:51:45.708656 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.708618 2562 generic.go:356] "Generic (PLEG): container finished" podID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" containerID="a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620" exitCode=0 Jul 09 19:51:45.709188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.708713 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686944fc45-jbxr9" Jul 09 19:51:45.709188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.708834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686944fc45-jbxr9" event={"ID":"aec928d9-81c2-4454-80dd-ccd650e2fcc2","Type":"ContainerDied","Data":"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620"} Jul 09 19:51:45.709188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.708874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686944fc45-jbxr9" event={"ID":"aec928d9-81c2-4454-80dd-ccd650e2fcc2","Type":"ContainerDied","Data":"47339382eb98489198eeae43525b392b76e19165d14b4c079731d5f962946762"} Jul 09 19:51:45.709188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.708898 2562 scope.go:117] "RemoveContainer" containerID="a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620" Jul 09 19:51:45.721372 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.721351 2562 scope.go:117] "RemoveContainer" containerID="a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620" Jul 09 19:51:45.721705 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:51:45.721683 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620\": container with ID starting with a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620 not found: ID does not exist" containerID="a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620" Jul 09 19:51:45.721781 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.721715 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620"} err="failed to get container status \"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620\": rpc error: code = NotFound desc = could not find container \"a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620\": container with ID starting with a3b16ff470773000606e182ff27a847a9a66cac436da44cce858d5450a875620 not found: ID does not exist" Jul 09 19:51:45.747716 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.747687 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:51:45.763953 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.763928 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-686944fc45-jbxr9"] Jul 09 19:51:45.953916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:51:45.953885 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" path="/var/lib/kubelet/pods/aec928d9-81c2-4454-80dd-ccd650e2fcc2/volumes" Jul 09 19:52:55.444263 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.444171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:52:55.541669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.541637 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:52:55.541892 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.541857 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" podUID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" containerName="manager" containerID="cri-o://2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb" gracePeriod=30 Jul 09 19:52:55.801265 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.801235 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:52:55.902394 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.902361 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert\") pod \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " Jul 09 19:52:55.902601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.902430 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqjz8\" (UniqueName: \"kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8\") pod \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\" (UID: \"ba32ed21-3e2e-4b56-b9b2-2c954033f50d\") " Jul 09 19:52:55.904414 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.904383 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ba32ed21-3e2e-4b56-b9b2-2c954033f50d" (UID: "ba32ed21-3e2e-4b56-b9b2-2c954033f50d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:52:55.904601 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:55.904572 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8" (OuterVolumeSpecName: "kube-api-access-dqjz8") pod "ba32ed21-3e2e-4b56-b9b2-2c954033f50d" (UID: "ba32ed21-3e2e-4b56-b9b2-2c954033f50d"). InnerVolumeSpecName "kube-api-access-dqjz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:52:56.003669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.003575 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-webhook-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:52:56.003669 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.003627 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqjz8\" (UniqueName: \"kubernetes.io/projected/ba32ed21-3e2e-4b56-b9b2-2c954033f50d-kube-api-access-dqjz8\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 19:52:56.109837 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.109804 2562 generic.go:356] "Generic (PLEG): container finished" podID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" containerID="2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb" exitCode=0 Jul 09 19:52:56.109990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.109849 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" Jul 09 19:52:56.109990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.109851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" event={"ID":"ba32ed21-3e2e-4b56-b9b2-2c954033f50d","Type":"ContainerDied","Data":"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb"} Jul 09 19:52:56.109990 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.109974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-45kfh" event={"ID":"ba32ed21-3e2e-4b56-b9b2-2c954033f50d","Type":"ContainerDied","Data":"27cbb9c75d48273d7e6b59cec1ea1326a65230f2fdf010ac5c30f5b3d83948bc"} Jul 09 19:52:56.110099 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.110008 2562 scope.go:117] "RemoveContainer" containerID="2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb" Jul 09 19:52:56.121699 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.121678 2562 scope.go:117] "RemoveContainer" containerID="2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb" Jul 09 19:52:56.121968 ip-10-0-139-128 kubenswrapper[2562]: E0709 19:52:56.121948 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb\": container with ID starting with 2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb not found: ID does not exist" containerID="2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb" Jul 09 19:52:56.122037 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.121977 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb"} err="failed to get container status \"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb\": rpc error: code = NotFound desc = could not find container \"2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb\": container with ID starting with 2bd571c4be427e3d6dad5a051e5a2d1d470ee2fd8e0f4f6cded7f2bd3fe3f0fb not found: ID does not exist" Jul 09 19:52:56.156267 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.156245 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:52:56.162850 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.162828 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-45kfh"] Jul 09 19:52:56.813992 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.813938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:52:56.924448 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.924408 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-sx24t"] Jul 09 19:52:56.924917 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.924898 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" containerName="authorino" Jul 09 19:52:56.924987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.924919 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" containerName="authorino" Jul 09 19:52:56.924987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.924932 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" containerName="manager" Jul 09 19:52:56.924987 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.924938 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" containerName="manager" Jul 09 19:52:56.925081 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.925002 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" containerName="manager" Jul 09 19:52:56.925081 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.925016 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec928d9-81c2-4454-80dd-ccd650e2fcc2" containerName="authorino" Jul 09 19:52:56.929188 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.929173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:56.933225 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.933205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 19:52:56.933225 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.933215 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 19:52:56.933418 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.933232 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r8qmr\"" Jul 09 19:52:56.937916 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:56.937898 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-sx24t"] Jul 09 19:52:57.012414 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.012384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ab6e2f-286f-40cc-860e-894515cd7a86-webhook-cert\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.012606 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.012470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7g9l\" (UniqueName: \"kubernetes.io/projected/d1ab6e2f-286f-40cc-860e-894515cd7a86-kube-api-access-v7g9l\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.113209 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.113114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ab6e2f-286f-40cc-860e-894515cd7a86-webhook-cert\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.113386 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.113232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7g9l\" (UniqueName: \"kubernetes.io/projected/d1ab6e2f-286f-40cc-860e-894515cd7a86-kube-api-access-v7g9l\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.115989 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.115960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1ab6e2f-286f-40cc-860e-894515cd7a86-webhook-cert\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.123718 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.123690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7g9l\" (UniqueName: \"kubernetes.io/projected/d1ab6e2f-286f-40cc-860e-894515cd7a86-kube-api-access-v7g9l\") pod \"maas-controller-6885c5f9d9-sx24t\" (UID: \"d1ab6e2f-286f-40cc-860e-894515cd7a86\") " pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.240608 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.240550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:57.383156 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.383123 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6885c5f9d9-sx24t"] Jul 09 19:52:57.384880 ip-10-0-139-128 kubenswrapper[2562]: W0709 19:52:57.384857 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ab6e2f_286f_40cc_860e_894515cd7a86.slice/crio-e2f037c597e754737bb8924049d208d95ef585a78f80db1d7f6d0490064aa9d9 WatchSource:0}: Error finding container e2f037c597e754737bb8924049d208d95ef585a78f80db1d7f6d0490064aa9d9: Status 404 returned error can't find the container with id e2f037c597e754737bb8924049d208d95ef585a78f80db1d7f6d0490064aa9d9 Jul 09 19:52:57.954081 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:57.954052 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba32ed21-3e2e-4b56-b9b2-2c954033f50d" path="/var/lib/kubelet/pods/ba32ed21-3e2e-4b56-b9b2-2c954033f50d/volumes" Jul 09 19:52:58.124338 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:58.124295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" event={"ID":"d1ab6e2f-286f-40cc-860e-894515cd7a86","Type":"ContainerStarted","Data":"16ec803437f6fb4e154fbfb5f24aeda9efa520866fa8eb0c1f5f2c8a890a7976"} Jul 09 19:52:58.124531 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:58.124368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" event={"ID":"d1ab6e2f-286f-40cc-860e-894515cd7a86","Type":"ContainerStarted","Data":"e2f037c597e754737bb8924049d208d95ef585a78f80db1d7f6d0490064aa9d9"} Jul 09 19:52:58.124647 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:58.124556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:52:58.144970 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:52:58.144916 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" podStartSLOduration=1.607563264 podStartE2EDuration="2.144901314s" totalImagesPullingTime="537.33805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:52:58.142442734 +0000 UTC m=+804.733202966" watchObservedRunningTime="2026-07-09 19:52:58.144901314 +0000 UTC m=+804.735661518" Jul 09 19:53:09.133354 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:09.133321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6885c5f9d9-sx24t" Jul 09 19:53:37.627290 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:37.627240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:37.729908 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:37.729872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:37.829483 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:37.829450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:43.556398 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:43.556356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:43.653135 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:43.653076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:43.756434 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:43.756400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:43.902539 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:43.902502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:49.733735 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:49.733698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:53:49.819888 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:53:49.819844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:54:07.861454 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:54:07.861407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:54:33.933511 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:54:33.933437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:54:33.938169 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:54:33.938147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:55:13.809230 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:55:13.809181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:55:15.121706 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:55:15.121670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:56:11.806331 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:56:11.806239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:56:11.904801 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:56:11.904767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 19:59:33.971768 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:59:33.971739 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 19:59:33.979739 ip-10-0-139-128 kubenswrapper[2562]: I0709 19:59:33.979707 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:00:00.147412 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.147370 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw"] Jul 09 20:00:00.151660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.151640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" Jul 09 20:00:00.154525 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.154501 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5hwxb\"" Jul 09 20:00:00.157078 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.157050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw"] Jul 09 20:00:00.163990 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.163973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" Jul 09 20:00:00.522549 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.522513 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw"] Jul 09 20:00:00.523133 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:00:00.523103 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ae6e11_9cae_41e5_9d2a_636065771db1.slice/crio-0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1 WatchSource:0}: Error finding container 0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1: Status 404 returned error can't find the container with id 0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1 Jul 09 20:00:00.525487 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:00.525465 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 20:00:01.456035 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:01.455995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" event={"ID":"50ae6e11-9cae-41e5-9d2a-636065771db1","Type":"ContainerStarted","Data":"0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1"} Jul 09 20:00:05.480723 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:05.480687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" event={"ID":"50ae6e11-9cae-41e5-9d2a-636065771db1","Type":"ContainerStarted","Data":"e6a9e615502541bbd1c3a11ab52c0c2880bbe1644df1b35abc79e508c060ad4c"} Jul 09 20:00:05.498196 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:05.498091 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" podStartSLOduration=1.115110807 podStartE2EDuration="5.498076549s" totalImagesPullingTime="4.382965742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:00:05.495635532 +0000 UTC m=+1232.086395752" watchObservedRunningTime="2026-07-09 20:00:05.498076549 +0000 UTC m=+1232.088836754" Jul 09 20:00:06.487858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:06.487819 2562 generic.go:356] "Generic (PLEG): container finished" podID="50ae6e11-9cae-41e5-9d2a-636065771db1" containerID="e6a9e615502541bbd1c3a11ab52c0c2880bbe1644df1b35abc79e508c060ad4c" exitCode=0 Jul 09 20:00:06.488247 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:06.487878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" event={"ID":"50ae6e11-9cae-41e5-9d2a-636065771db1","Type":"ContainerDied","Data":"e6a9e615502541bbd1c3a11ab52c0c2880bbe1644df1b35abc79e508c060ad4c"} Jul 09 20:00:07.633808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:07.633782 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" Jul 09 20:00:08.499705 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:08.499676 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" Jul 09 20:00:08.499886 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:08.499720 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727120-b28rw" event={"ID":"50ae6e11-9cae-41e5-9d2a-636065771db1","Type":"ContainerDied","Data":"0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1"} Jul 09 20:00:08.499886 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:08.499776 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c700bc3574d44b0b32af868ce1ad1fce4cfa949b73bc56a31994178f3388be1" Jul 09 20:00:28.323922 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:00:28.323832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:01:07.042702 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:01:07.042665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:02:52.928523 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:02:52.928489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:02:52.976500 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:02:52.976466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:04:34.006879 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:04:34.006765 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:04:34.017084 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:04:34.017060 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:04:42.645396 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:04:42.645354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:04:49.632531 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:04:49.632476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:04:57.901561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:04:57.901475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:03.035960 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:03.035920 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:05.876158 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:05.876118 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:05.977132 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:05.977097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:06.331645 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:06.331602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:12.159835 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.159800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:12.265480 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.265446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:12.270791 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.270764 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:05:12.271181 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.271169 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ae6e11-9cae-41e5-9d2a-636065771db1" containerName="cleanup" Jul 09 20:05:12.271240 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.271182 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ae6e11-9cae-41e5-9d2a-636065771db1" containerName="cleanup" Jul 09 20:05:12.271240 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.271239 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ae6e11-9cae-41e5-9d2a-636065771db1" containerName="cleanup" Jul 09 20:05:12.273911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.273891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.276859 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.276841 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-581685c1-openshift-default-dockercfg-tfbmk\"" Jul 09 20:05:12.295496 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.295452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:05:12.363202 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.363150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:12.454148 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454148 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454476 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454476 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.454476 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.454401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbkn\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.554957 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.554920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.554957 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.554963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.554986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbkn\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555564 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555564 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555564 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555710 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.555919 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.555897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.557326 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.557306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.557850 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.557831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.566779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.566757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.566909 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.566760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbkn\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn\") pod \"e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.586733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.586710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:12.729717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.729688 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:05:12.730222 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:05:12.730196 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e1eab2_8d8f_48d7_8081_04751752e86c.slice/crio-fc4fa1c5f2640a43a7ade013f1bb222f6a1a32e7dbefaf52d5c223ce3a22f3a0 WatchSource:0}: Error finding container fc4fa1c5f2640a43a7ade013f1bb222f6a1a32e7dbefaf52d5c223ce3a22f3a0: Status 404 returned error can't find the container with id fc4fa1c5f2640a43a7ade013f1bb222f6a1a32e7dbefaf52d5c223ce3a22f3a0 Jul 09 20:05:12.732269 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.732234 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:12.732370 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.732333 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:12.732434 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:12.732383 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:13.209782 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.209744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:13.252442 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.252405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerStarted","Data":"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f"} Jul 09 20:05:13.252646 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.252481 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerStarted","Data":"fc4fa1c5f2640a43a7ade013f1bb222f6a1a32e7dbefaf52d5c223ce3a22f3a0"} Jul 09 20:05:13.273142 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.273092 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podStartSLOduration=1.273075566 podStartE2EDuration="1.273075566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:05:13.271098903 +0000 UTC m=+1539.861859128" watchObservedRunningTime="2026-07-09 20:05:13.273075566 +0000 UTC m=+1539.863835770" Jul 09 20:05:13.394472 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.394432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:13.466391 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.466294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:13.528111 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.528072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:13.587307 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.587269 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:13.593721 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.593693 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:13.638139 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.638104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:13.953616 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:13.953568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:14.054655 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:14.054625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:14.259093 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:14.259017 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:14.259962 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:14.259943 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:15.982045 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:15.982015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:05:18.578826 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.578794 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:18.582418 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.582401 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:18.585010 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.584975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5hwxb\"" Jul 09 20:05:18.588798 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.588772 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:18.613057 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.613031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-4kpzh\" (UID: \"d2f65e2d-17b3-45d5-91fe-94a84144fd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:18.713641 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.713606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-4kpzh\" (UID: \"d2f65e2d-17b3-45d5-91fe-94a84144fd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:18.714248 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.714226 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-4kpzh\" (UID: \"d2f65e2d-17b3-45d5-91fe-94a84144fd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:18.896021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:18.895990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:19.086124 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:19.086096 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:19.088290 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:05:19.088262 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f65e2d_17b3_45d5_91fe_94a84144fd8d.slice/crio-de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3 WatchSource:0}: Error finding container de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3: Status 404 returned error can't find the container with id de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3 Jul 09 20:05:19.289496 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:19.289460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerStarted","Data":"de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3"} Jul 09 20:05:20.298821 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:20.298727 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerID="c0de791e89e81579451b37538dfa29488642ca3844a50a78c85a55ec905fa3d4" exitCode=7 Jul 09 20:05:20.299172 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:20.298833 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerDied","Data":"c0de791e89e81579451b37538dfa29488642ca3844a50a78c85a55ec905fa3d4"} Jul 09 20:05:20.299172 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:20.299074 2562 scope.go:117] "RemoveContainer" containerID="c0de791e89e81579451b37538dfa29488642ca3844a50a78c85a55ec905fa3d4" Jul 09 20:05:21.306759 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:21.306721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerStarted","Data":"46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a"} Jul 09 20:05:21.322147 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:21.322095 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" podStartSLOduration=3.322081236 podStartE2EDuration="3.322081236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:05:21.321138593 +0000 UTC m=+1547.911898797" watchObservedRunningTime="2026-07-09 20:05:21.322081236 +0000 UTC m=+1547.912841440" Jul 09 20:05:22.313342 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:22.313307 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerID="46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a" exitCode=7 Jul 09 20:05:22.313788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:22.313386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerDied","Data":"46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a"} Jul 09 20:05:22.313788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:22.313447 2562 scope.go:117] "RemoveContainer" containerID="c0de791e89e81579451b37538dfa29488642ca3844a50a78c85a55ec905fa3d4" Jul 09 20:05:22.313788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:22.313573 2562 scope.go:117] "RemoveContainer" containerID="46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a" Jul 09 20:05:22.313912 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:05:22.313837 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-581685c1-4kpzh_odh-ai-gateway-infra(d2f65e2d-17b3-45d5-91fe-94a84144fd8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" Jul 09 20:05:23.321603 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:23.321544 2562 scope.go:117] "RemoveContainer" containerID="46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a" Jul 09 20:05:23.322066 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:05:23.321821 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-581685c1-4kpzh_odh-ai-gateway-infra(d2f65e2d-17b3-45d5-91fe-94a84144fd8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" Jul 09 20:05:25.333930 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.333828 2562 generic.go:356] "Generic (PLEG): container finished" podID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerID="35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f" exitCode=0 Jul 09 20:05:25.333930 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.333910 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerDied","Data":"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f"} Jul 09 20:05:25.334370 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.334285 2562 scope.go:117] "RemoveContainer" containerID="35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f" Jul 09 20:05:25.335230 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.335201 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:25.335361 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.335283 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:25.335361 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:25.335329 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:05:26.352615 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:26.351842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerStarted","Data":"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5"} Jul 09 20:05:26.587375 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:26.587331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:26.588654 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:26.588627 2562 patch_prober.go:28] interesting pod/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 09 20:05:26.588770 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:26.588681 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 09 20:05:27.587867 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:27.587833 2562 patch_prober.go:28] interesting pod/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 09 20:05:27.588249 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:27.587889 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 09 20:05:28.588079 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:28.588045 2562 patch_prober.go:28] interesting pod/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 09 20:05:28.588465 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:28.588099 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 09 20:05:29.259380 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:29.259346 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:29.587718 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:29.587631 2562 patch_prober.go:28] interesting pod/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 09 20:05:29.587718 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:29.587693 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 09 20:05:30.590747 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:30.590718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:30.591804 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:30.591780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:05:35.954313 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:35.954281 2562 scope.go:117] "RemoveContainer" containerID="46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a" Jul 09 20:05:36.419846 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:36.419811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerStarted","Data":"380a0b186f8c25dfacdfa2bee4bbaeffb5d206476a88bdef7986eee4bbea6376"} Jul 09 20:05:37.428035 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:37.428003 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerID="380a0b186f8c25dfacdfa2bee4bbaeffb5d206476a88bdef7986eee4bbea6376" exitCode=0 Jul 09 20:05:37.428555 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:37.428078 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerDied","Data":"380a0b186f8c25dfacdfa2bee4bbaeffb5d206476a88bdef7986eee4bbea6376"} Jul 09 20:05:37.428555 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:37.428145 2562 scope.go:117] "RemoveContainer" containerID="46bfe19b6bc9d4cd0abf1afbe2f222b62f2808903d9d08d2bb852fc440214f9a" Jul 09 20:05:37.444217 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:37.444193 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:38.575140 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:38.575110 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:38.721749 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:38.721660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca\") pod \"d2f65e2d-17b3-45d5-91fe-94a84144fd8d\" (UID: \"d2f65e2d-17b3-45d5-91fe-94a84144fd8d\") " Jul 09 20:05:38.722025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:38.722003 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2f65e2d-17b3-45d5-91fe-94a84144fd8d" (UID: "d2f65e2d-17b3-45d5-91fe-94a84144fd8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:05:38.822974 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:38.822937 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f65e2d-17b3-45d5-91fe-94a84144fd8d-maas-api-service-ca\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:05:39.442436 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.442410 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" Jul 09 20:05:39.442619 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.442448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh" event={"ID":"d2f65e2d-17b3-45d5-91fe-94a84144fd8d","Type":"ContainerDied","Data":"de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3"} Jul 09 20:05:39.442619 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.442502 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6f02a2150f3e78b0b6ea61dabb62b41bb1dc111b2d40b0ae3688afb11860e3" Jul 09 20:05:39.472699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.472674 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:39.474911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.474892 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-4kpzh"] Jul 09 20:05:39.954094 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:05:39.954058 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" path="/var/lib/kubelet/pods/d2f65e2d-17b3-45d5-91fe-94a84144fd8d/volumes" Jul 09 20:06:18.620091 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620055 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx"] Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620447 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620457 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620471 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620477 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620541 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.620561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.620552 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:18.623723 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.623697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:18.626247 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.626221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5hwxb\"" Jul 09 20:06:18.631364 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.631341 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx"] Jul 09 20:06:18.710234 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.710194 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:06:18.710469 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.710423 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" containerID="cri-o://84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5" gracePeriod=30 Jul 09 20:06:18.733420 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.733379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:18.771927 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.771892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-p5nhx\" (UID: \"0bba33cd-e80b-46c3-82e4-52665f9ff25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:18.872601 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.872498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-p5nhx\" (UID: \"0bba33cd-e80b-46c3-82e4-52665f9ff25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:18.873175 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.873154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-581685c1-p5nhx\" (UID: \"0bba33cd-e80b-46c3-82e4-52665f9ff25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:18.937131 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:18.937094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:19.085155 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:19.085115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx"] Jul 09 20:06:19.086857 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:06:19.086829 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bba33cd_e80b_46c3_82e4_52665f9ff25a.slice/crio-0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f WatchSource:0}: Error finding container 0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f: Status 404 returned error can't find the container with id 0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f Jul 09 20:06:19.666526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:19.666495 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" event={"ID":"0bba33cd-e80b-46c3-82e4-52665f9ff25a","Type":"ContainerStarted","Data":"46216853b47a4c6e7566b4b0fb6fd6ccfa5bffaa3e22369127b98091885375dc"} Jul 09 20:06:19.666954 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:19.666547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" event={"ID":"0bba33cd-e80b-46c3-82e4-52665f9ff25a","Type":"ContainerStarted","Data":"0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f"} Jul 09 20:06:19.682530 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:19.682483 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" podStartSLOduration=1.682468435 podStartE2EDuration="1.682468435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:06:19.680890561 +0000 UTC m=+1606.271650793" watchObservedRunningTime="2026-07-09 20:06:19.682468435 +0000 UTC m=+1606.273228662" Jul 09 20:06:19.903482 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:19.903455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:20.680715 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:20.680673 2562 generic.go:356] "Generic (PLEG): container finished" podID="0bba33cd-e80b-46c3-82e4-52665f9ff25a" containerID="46216853b47a4c6e7566b4b0fb6fd6ccfa5bffaa3e22369127b98091885375dc" exitCode=0 Jul 09 20:06:20.681070 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:20.680761 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" event={"ID":"0bba33cd-e80b-46c3-82e4-52665f9ff25a","Type":"ContainerDied","Data":"46216853b47a4c6e7566b4b0fb6fd6ccfa5bffaa3e22369127b98091885375dc"} Jul 09 20:06:21.822985 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:21.822961 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:21.901763 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:21.901728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca\") pod \"0bba33cd-e80b-46c3-82e4-52665f9ff25a\" (UID: \"0bba33cd-e80b-46c3-82e4-52665f9ff25a\") " Jul 09 20:06:21.902072 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:21.902049 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bba33cd-e80b-46c3-82e4-52665f9ff25a" (UID: "0bba33cd-e80b-46c3-82e4-52665f9ff25a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:06:22.002733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:22.002708 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bba33cd-e80b-46c3-82e4-52665f9ff25a-maas-api-service-ca\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:22.693277 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:22.693249 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" Jul 09 20:06:22.693486 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:22.693244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-581685c1-p5nhx" event={"ID":"0bba33cd-e80b-46c3-82e4-52665f9ff25a","Type":"ContainerDied","Data":"0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f"} Jul 09 20:06:22.693486 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:22.693385 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fafa4a41532d49cf6d95eabaed4ed65bed0270814f41f048d1aae8b7515bb4f" Jul 09 20:06:23.978570 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:23.978512 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:06:24.122879 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.122847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.122905 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.122941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.122966 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123004 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123287 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123071 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123287 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmbkn\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123287 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123133 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123287 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123166 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data\") pod \"39e1eab2-8d8f-48d7-8081-04751752e86c\" (UID: \"39e1eab2-8d8f-48d7-8081-04751752e86c\") " Jul 09 20:06:24.123287 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:06:24.123527 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123469 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.123622 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123560 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:06:24.123735 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123708 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:06:24.123891 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.123869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:06:24.124228 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.124205 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data" (OuterVolumeSpecName: "istio-data") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:06:24.125232 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.125203 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:06:24.125321 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.125260 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn" (OuterVolumeSpecName: "kube-api-access-hmbkn") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "kube-api-access-hmbkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:06:24.125692 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.125675 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:06:24.125790 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.125768 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token" (OuterVolumeSpecName: "istio-token") pod "39e1eab2-8d8f-48d7-8081-04751752e86c" (UID: "39e1eab2-8d8f-48d7-8081-04751752e86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:06:24.224322 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224273 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbkn\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-kube-api-access-hmbkn\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224322 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224315 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e1eab2-8d8f-48d7-8081-04751752e86c-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224322 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224328 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224340 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224352 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224362 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224372 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e1eab2-8d8f-48d7-8081-04751752e86c-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.224563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.224383 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e1eab2-8d8f-48d7-8081-04751752e86c-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:06:24.484818 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.484781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:24.585240 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.585140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:24.706781 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.706744 2562 generic.go:356] "Generic (PLEG): container finished" podID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerID="84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5" exitCode=0 Jul 09 20:06:24.706925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.706798 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" Jul 09 20:06:24.706925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.706827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerDied","Data":"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5"} Jul 09 20:06:24.706925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.706879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7" event={"ID":"39e1eab2-8d8f-48d7-8081-04751752e86c","Type":"ContainerDied","Data":"fc4fa1c5f2640a43a7ade013f1bb222f6a1a32e7dbefaf52d5c223ce3a22f3a0"} Jul 09 20:06:24.706925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.706903 2562 scope.go:117] "RemoveContainer" containerID="84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5" Jul 09 20:06:24.717221 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.717184 2562 scope.go:117] "RemoveContainer" containerID="35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f" Jul 09 20:06:24.733214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.733178 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:06:24.733969 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.733954 2562 scope.go:117] "RemoveContainer" containerID="84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5" Jul 09 20:06:24.734243 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:06:24.734222 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5\": container with ID starting with 84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5 not found: ID does not exist" containerID="84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5" Jul 09 20:06:24.734309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.734253 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5"} err="failed to get container status \"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5\": rpc error: code = NotFound desc = could not find container \"84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5\": container with ID starting with 84e58fc390dfb185555d8702f1ee7b55396e5795aa21e72d91b28a558014f5d5 not found: ID does not exist" Jul 09 20:06:24.734309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.734284 2562 scope.go:117] "RemoveContainer" containerID="35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f" Jul 09 20:06:24.734524 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:06:24.734506 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f\": container with ID starting with 35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f not found: ID does not exist" containerID="35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f" Jul 09 20:06:24.734566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.734530 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f"} err="failed to get container status \"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f\": rpc error: code = NotFound desc = could not find container \"35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f\": container with ID starting with 35040c19438326bfad3f1df4f658fe6a602426cf24f77d5c6851becada35b47f not found: ID does not exist" Jul 09 20:06:24.739036 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:24.738981 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-581685c1-openshift-default-74fdffcc49-xnrf7"] Jul 09 20:06:25.142835 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:25.142799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:25.954167 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:25.954135 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" path="/var/lib/kubelet/pods/39e1eab2-8d8f-48d7-8081-04751752e86c/volumes" Jul 09 20:06:29.897413 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:29.897386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:29.998991 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:29.998962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:30.083628 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.083598 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:06:30.084304 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084275 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.084304 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084305 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084317 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bba33cd-e80b-46c3-82e4-52665f9ff25a" containerName="revoke-keys" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084325 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bba33cd-e80b-46c3-82e4-52665f9ff25a" containerName="revoke-keys" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084350 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084358 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084368 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084376 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:30.084499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084491 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.084856 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084503 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bba33cd-e80b-46c3-82e4-52665f9ff25a" containerName="revoke-keys" Jul 09 20:06:30.084856 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084513 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f65e2d-17b3-45d5-91fe-94a84144fd8d" containerName="revoke-keys" Jul 09 20:06:30.084856 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.084528 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e1eab2-8d8f-48d7-8081-04751752e86c" containerName="istio-proxy" Jul 09 20:06:30.089201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.089173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.091818 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.091798 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8c04d3b2-openshift-default-dockercfg-7smw8\"" Jul 09 20:06:30.102132 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.102108 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:06:30.120122 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.120101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:30.173806 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.173736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:30.180023 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.179997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180118 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180118 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnw4\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180195 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180195 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180285 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180285 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180375 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.180375 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.180311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281269 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281766 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281766 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281766 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnw4\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.281766 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.281741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.282041 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.282019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.282112 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.282029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.282112 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.282082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.282557 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.282532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.283702 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.283676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.283948 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.283931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.289823 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.289800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.289908 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.289863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnw4\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4\") pod \"e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.400763 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.400726 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:30.540854 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.540711 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:06:30.545062 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.544725 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:30.545062 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.544861 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:30.545062 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.544944 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:30.744521 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.744426 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerStarted","Data":"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89"} Jul 09 20:06:30.744521 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.744480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerStarted","Data":"7f02da477afb863fec2fcd24436207ccc0bf813f0a171d719843dbfa58136a23"} Jul 09 20:06:30.765807 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:30.765756 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podStartSLOduration=0.765740196 podStartE2EDuration="765.740196ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:06:30.76437492 +0000 UTC m=+1617.355135152" watchObservedRunningTime="2026-07-09 20:06:30.765740196 +0000 UTC m=+1617.356500400" Jul 09 20:06:31.223184 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.223146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:31.300189 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.300156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:31.365720 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.365684 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:31.401826 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.401795 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:31.416335 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.416306 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:31.637592 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.637559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:31.750330 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.750304 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:31.751428 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.751411 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:31.789453 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.789423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:31.887695 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:31.887568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:33.590760 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:33.590731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:06:42.822402 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.822366 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6248434-2b95-40e5-b265-66b5a22d929d" containerID="cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89" exitCode=0 Jul 09 20:06:42.822929 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.822428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerDied","Data":"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89"} Jul 09 20:06:42.822929 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.822819 2562 scope.go:117] "RemoveContainer" containerID="cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89" Jul 09 20:06:42.823826 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.823793 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:42.823896 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.823880 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:42.823953 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:42.823937 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:06:43.832004 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:43.831960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerStarted","Data":"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63"} Jul 09 20:06:44.400896 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:44.400866 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:44.402167 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:44.402139 2562 patch_prober.go:28] interesting pod/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 09 20:06:44.402317 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:44.402206 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 09 20:06:45.401529 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:45.401494 2562 patch_prober.go:28] interesting pod/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 09 20:06:45.401944 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:45.401562 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 09 20:06:46.401656 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:46.401619 2562 patch_prober.go:28] interesting pod/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 09 20:06:46.402023 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:46.401673 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 09 20:06:46.750355 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:46.750226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:47.402084 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:47.402050 2562 patch_prober.go:28] interesting pod/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 09 20:06:47.402478 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:47.402097 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 09 20:06:48.405684 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:48.405656 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:06:48.406929 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:06:48.406906 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:07:00.374605 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:00.374545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:06.775709 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:06.775677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:11.471955 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:11.471919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:17.157670 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:17.157634 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:17.263928 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:17.263891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:22.505446 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.505409 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:07:22.505880 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.505638 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" containerID="cri-o://041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63" gracePeriod=30 Jul 09 20:07:22.529328 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.529292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:22.809951 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.809867 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:07:22.812512 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.812491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.814998 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.814981 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e2f7b0b4-openshift-default-dockercfg-kkq7h\"" Jul 09 20:07:22.822608 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.822570 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:07:22.844510 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.844486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:22.854733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854937 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854937 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854937 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.854937 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.855064 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.854947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955316 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955506 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955506 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955651 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955710 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955768 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955768 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955768 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955768 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.955980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.955966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.956383 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.956357 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.958341 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.958317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.958494 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.958479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.963652 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.963623 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:22.963770 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:22.963753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp\") pod \"e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:23.125644 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.125526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:23.263565 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.263533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:07:23.265057 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:07:23.265026 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac27be2_1d6f_46f6_845a_fc3f4fb369ac.slice/crio-f81885363b7e80250afe7584ad2cf0c830886d7ba1ac728a50da5bc6ce3fac8e WatchSource:0}: Error finding container f81885363b7e80250afe7584ad2cf0c830886d7ba1ac728a50da5bc6ce3fac8e: Status 404 returned error can't find the container with id f81885363b7e80250afe7584ad2cf0c830886d7ba1ac728a50da5bc6ce3fac8e Jul 09 20:07:23.267553 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.267508 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:23.267653 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.267632 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:23.267707 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.267672 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:23.952556 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:23.952516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:24.049205 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.049159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:24.073233 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.073197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerStarted","Data":"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59"} Jul 09 20:07:24.073439 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.073269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerStarted","Data":"f81885363b7e80250afe7584ad2cf0c830886d7ba1ac728a50da5bc6ce3fac8e"} Jul 09 20:07:24.093519 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.093469 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" podStartSLOduration=2.09345425 podStartE2EDuration="2.09345425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:24.091292235 +0000 UTC m=+1670.682052464" watchObservedRunningTime="2026-07-09 20:07:24.09345425 +0000 UTC m=+1670.684214455" Jul 09 20:07:24.126492 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.126454 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:24.132026 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.131987 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:24.235304 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.235221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:24.469684 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.469646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:24.640369 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.640324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:24.742734 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:24.742696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:25.079252 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:25.079166 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:25.080142 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:25.080121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:27.806651 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.806626 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:07:27.901755 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901722 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.901932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.901932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901813 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.901932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.901932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901897 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.901932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901923 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clnw4\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.902309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901946 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.902309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.901967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.902309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy\") pod \"c6248434-2b95-40e5-b265-66b5a22d929d\" (UID: \"c6248434-2b95-40e5-b265-66b5a22d929d\") " Jul 09 20:07:27.902309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902029 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:07:27.902309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902278 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:07:27.902597 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902327 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:27.902597 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902334 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:07:27.902597 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902469 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:07:27.902875 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.902849 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:07:27.904269 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.904243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:07:27.904507 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.904472 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4" (OuterVolumeSpecName: "kube-api-access-clnw4") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "kube-api-access-clnw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:07:27.904909 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.904884 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:07:27.905012 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:27.904913 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6248434-2b95-40e5-b265-66b5a22d929d" (UID: "c6248434-2b95-40e5-b265-66b5a22d929d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:07:28.003419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003390 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003415 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003423 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003431 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6248434-2b95-40e5-b265-66b5a22d929d-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003439 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6248434-2b95-40e5-b265-66b5a22d929d-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003447 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnw4\" (UniqueName: \"kubernetes.io/projected/c6248434-2b95-40e5-b265-66b5a22d929d-kube-api-access-clnw4\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003457 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.003660 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.003465 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6248434-2b95-40e5-b265-66b5a22d929d-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:07:28.097131 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.097040 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6248434-2b95-40e5-b265-66b5a22d929d" containerID="041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63" exitCode=0 Jul 09 20:07:28.097131 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.097090 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" Jul 09 20:07:28.097347 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.097132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerDied","Data":"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63"} Jul 09 20:07:28.097347 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.097196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z" event={"ID":"c6248434-2b95-40e5-b265-66b5a22d929d","Type":"ContainerDied","Data":"7f02da477afb863fec2fcd24436207ccc0bf813f0a171d719843dbfa58136a23"} Jul 09 20:07:28.097347 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.097219 2562 scope.go:117] "RemoveContainer" containerID="041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63" Jul 09 20:07:28.108893 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.108869 2562 scope.go:117] "RemoveContainer" containerID="cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89" Jul 09 20:07:28.116430 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.116397 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:07:28.119046 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.119032 2562 scope.go:117] "RemoveContainer" containerID="041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63" Jul 09 20:07:28.119301 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:07:28.119285 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63\": container with ID starting with 041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63 not found: ID does not exist" containerID="041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63" Jul 09 20:07:28.119352 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.119309 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63"} err="failed to get container status \"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63\": rpc error: code = NotFound desc = could not find container \"041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63\": container with ID starting with 041317ce1b38f37c2e0e68a8ad856bc33bc81fb485e53a65da2d6507ad45bc63 not found: ID does not exist" Jul 09 20:07:28.119352 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.119328 2562 scope.go:117] "RemoveContainer" containerID="cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89" Jul 09 20:07:28.119549 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:07:28.119534 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89\": container with ID starting with cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89 not found: ID does not exist" containerID="cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89" Jul 09 20:07:28.119610 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.119552 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89"} err="failed to get container status \"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89\": rpc error: code = NotFound desc = could not find container \"cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89\": container with ID starting with cfa848a50af46c346ee3029936f8c14993e5bbf62035891c0e489d362b069b89 not found: ID does not exist" Jul 09 20:07:28.119904 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:28.119887 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8c04d3b2-openshift-default-c6c95f48c-qrn4z"] Jul 09 20:07:29.095517 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:29.095489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:29.196421 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:29.196390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:29.956295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:29.956258 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" path="/var/lib/kubelet/pods/c6248434-2b95-40e5-b265-66b5a22d929d/volumes" Jul 09 20:07:32.431564 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:32.431523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:07:35.140416 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.140380 2562 generic.go:356] "Generic (PLEG): container finished" podID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerID="f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59" exitCode=0 Jul 09 20:07:35.140904 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.140458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerDied","Data":"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59"} Jul 09 20:07:35.140904 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.140874 2562 scope.go:117] "RemoveContainer" containerID="f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59" Jul 09 20:07:35.141866 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.141837 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:35.141982 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.141922 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:35.141982 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:35.141971 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:07:36.152140 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:36.152104 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerStarted","Data":"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8"} Jul 09 20:07:37.126765 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:37.126733 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:37.127827 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:37.127798 2562 patch_prober.go:28] interesting pod/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 09 20:07:37.128056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:37.128029 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 09 20:07:38.126952 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:38.126916 2562 patch_prober.go:28] interesting pod/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 09 20:07:38.127527 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:38.127494 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 09 20:07:39.126162 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:39.126114 2562 patch_prober.go:28] interesting pod/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 09 20:07:39.126395 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:39.126185 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 09 20:07:40.079769 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:40.079735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:40.140762 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:40.140727 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:07:40.178853 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:07:40.178825 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:08:19.380184 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:19.380145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:19.478991 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:19.478962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:28.494065 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:28.494027 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:08:28.494536 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:28.494278 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" containerID="cri-o://764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8" gracePeriod=30 Jul 09 20:08:28.520245 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:28.520212 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:28.631167 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:28.631140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:28.919502 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:28.919458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:29.020343 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:29.020308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:33.745075 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.745052 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:08:33.818570 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818533 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818688 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818722 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818947 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818748 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818947 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818775 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818947 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818850 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.818947 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.818898 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket\") pod \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\" (UID: \"aac27be2-1d6f-46f6-845a-fc3f4fb369ac\") " Jul 09 20:08:33.819169 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:08:33.819169 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819156 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:08:33.819275 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:08:33.819275 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819248 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.819275 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819269 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.819394 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:08:33.819568 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.819538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:08:33.821119 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.821093 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:08:33.821397 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.821374 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp" (OuterVolumeSpecName: "kube-api-access-nj8lp") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "kube-api-access-nj8lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:08:33.821802 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.821776 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:08:33.821802 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.821780 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aac27be2-1d6f-46f6-845a-fc3f4fb369ac" (UID: "aac27be2-1d6f-46f6-845a-fc3f4fb369ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:08:33.920037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920007 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920031 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-kube-api-access-nj8lp\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920040 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920048 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920056 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920064 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:33.920262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:33.920071 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac27be2-1d6f-46f6-845a-fc3f4fb369ac-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:08:34.380110 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.380075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:34.480269 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.480239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:34.498118 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.498087 2562 generic.go:356] "Generic (PLEG): container finished" podID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerID="764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8" exitCode=0 Jul 09 20:08:34.498299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.498151 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" Jul 09 20:08:34.498299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.498172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerDied","Data":"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8"} Jul 09 20:08:34.498299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.498248 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc" event={"ID":"aac27be2-1d6f-46f6-845a-fc3f4fb369ac","Type":"ContainerDied","Data":"f81885363b7e80250afe7584ad2cf0c830886d7ba1ac728a50da5bc6ce3fac8e"} Jul 09 20:08:34.498299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.498270 2562 scope.go:117] "RemoveContainer" containerID="764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8" Jul 09 20:08:34.507823 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.507806 2562 scope.go:117] "RemoveContainer" containerID="f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59" Jul 09 20:08:34.516656 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.516637 2562 scope.go:117] "RemoveContainer" containerID="764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8" Jul 09 20:08:34.516922 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:08:34.516905 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8\": container with ID starting with 764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8 not found: ID does not exist" containerID="764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8" Jul 09 20:08:34.516971 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.516930 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8"} err="failed to get container status \"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8\": rpc error: code = NotFound desc = could not find container \"764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8\": container with ID starting with 764d73cf17162a7ddfaa3fd2ad564d4199f615b4e5d32f7d58335840e6f85ef8 not found: ID does not exist" Jul 09 20:08:34.516971 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.516948 2562 scope.go:117] "RemoveContainer" containerID="f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59" Jul 09 20:08:34.517144 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:08:34.517130 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59\": container with ID starting with f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59 not found: ID does not exist" containerID="f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59" Jul 09 20:08:34.517199 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.517149 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59"} err="failed to get container status \"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59\": rpc error: code = NotFound desc = could not find container \"f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59\": container with ID starting with f9fedbbe8647e26ff0988b10eb88a3c95873b28af3e02eebc451383bbd90af59 not found: ID does not exist" Jul 09 20:08:34.517710 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.517691 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:08:34.522302 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:34.522281 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e2f7b0b4-openshift-default-867f4cfdc5-ktcbc"] Jul 09 20:08:35.134412 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:35.134374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:35.954063 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:35.954029 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" path="/var/lib/kubelet/pods/aac27be2-1d6f-46f6-845a-fc3f4fb369ac/volumes" Jul 09 20:08:39.789083 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:39.788988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:39.863493 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:39.863461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:40.300144 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:40.300109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:40.401879 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:40.401844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:40.510922 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:40.510884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:42.115313 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:42.115282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:47.872228 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:47.872190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:47.939670 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:47.939626 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:48.067657 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:48.067626 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:48.203344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:48.203312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:48.292159 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:48.292123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:08:48.683876 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:08:48.683838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:14.722461 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:14.722426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:20.433849 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:20.433807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:20.604139 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:20.604101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:20.694999 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:20.694912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:34.042806 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:34.042776 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:09:34.055458 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:34.055437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:09:36.616177 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:36.616145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:42.399138 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:42.399103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:42.455718 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:42.455684 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:42.549963 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:42.549921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:42.649983 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:42.649889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:53.178945 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:53.178904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:54.007504 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:54.007470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:59.798323 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:59.798290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:59.863609 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:59.863556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:09:59.943658 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:09:59.943628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:00.042193 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:00.042157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:00.142111 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:00.142077 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:00.899689 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:00.899654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:01.012554 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:01.012520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:01.109196 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:01.109151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:02.783639 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:02.783604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:08.315542 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:08.315489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:08.415921 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:08.415853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:09.013177 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:09.013147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.263304 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.263235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.363535 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.363481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.509970 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.509934 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.813349 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813319 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:10:15.813758 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813743 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:10:15.813816 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813759 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:10:15.813816 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813772 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813816 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813777 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813816 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813791 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813816 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813797 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813986 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813868 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813986 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813877 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6248434-2b95-40e5-b265-66b5a22d929d" containerName="istio-proxy" Jul 09 20:10:15.813986 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813883 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:10:15.813986 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.813889 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:10:15.816932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.816916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.819755 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.819734 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9bad9204-openshift-default-dockercfg-c6zhj\"" Jul 09 20:10:15.838625 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.838602 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:10:15.841447 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.841427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.896717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.896717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.896717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q67b\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.897048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.896877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.961601 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.961548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:15.997915 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.997887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998020 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.997920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998020 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.997946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998102 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998150 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q67b\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998269 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998339 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998414 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998414 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998540 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998540 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998686 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.998686 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.998651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:15.999108 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:15.999056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.000484 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.000463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.000812 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.000796 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.014426 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.014407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q67b\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.014681 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.014661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token\") pod \"e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.128061 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.127964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:16.274904 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.274874 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:10:16.276285 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:10:16.276257 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa6cd8e_53e0_45fc_a744_f85b7065a40e.slice/crio-b1d766677698dbf4f23d421bbf346ae0dfa10578f9264276b0a5d17c7d8e8695 WatchSource:0}: Error finding container b1d766677698dbf4f23d421bbf346ae0dfa10578f9264276b0a5d17c7d8e8695: Status 404 returned error can't find the container with id b1d766677698dbf4f23d421bbf346ae0dfa10578f9264276b0a5d17c7d8e8695 Jul 09 20:10:16.278341 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.278310 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:16.278414 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.278379 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:16.278414 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:16.278408 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:17.080985 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:17.080949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerStarted","Data":"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2"} Jul 09 20:10:17.081163 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:17.081002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerStarted","Data":"b1d766677698dbf4f23d421bbf346ae0dfa10578f9264276b0a5d17c7d8e8695"} Jul 09 20:10:17.102444 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:17.102397 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podStartSLOduration=2.102380658 podStartE2EDuration="2.102380658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:10:17.100907608 +0000 UTC m=+1843.691667835" watchObservedRunningTime="2026-07-09 20:10:17.102380658 +0000 UTC m=+1843.693140863" Jul 09 20:10:17.128397 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:17.128369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:17.133993 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:17.133968 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:18.087181 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:18.087151 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:18.088212 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:18.088190 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:19.563759 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:19.563726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:19.652208 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:19.652178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:19.714730 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:19.714700 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:19.771804 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:19.771776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:20.270991 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:20.270953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:20.371930 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:20.371890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:21.348247 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:21.348205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:23.989496 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:23.989463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:24.684539 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:24.684503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:24.785329 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:24.785295 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:25.155690 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:25.155649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:30.827318 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:30.827282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:30.883868 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:30.883831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:34.128772 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:34.128731 2562 patch_prober.go:28] interesting pod/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 20:10:34.129217 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:34.128801 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 20:10:35.193160 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.193126 2562 generic.go:356] "Generic (PLEG): container finished" podID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerID="ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2" exitCode=0 Jul 09 20:10:35.193616 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.193211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerDied","Data":"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2"} Jul 09 20:10:35.193850 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.193645 2562 scope.go:117] "RemoveContainer" containerID="ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2" Jul 09 20:10:35.194743 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.194717 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:35.194826 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.194784 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:35.194864 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:35.194823 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:10:36.200347 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:36.200305 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerStarted","Data":"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d"} Jul 09 20:10:37.129121 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:37.129090 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:37.129544 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:37.129462 2562 patch_prober.go:28] interesting pod/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 09 20:10:37.129746 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:37.129721 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 09 20:10:38.128949 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:38.128909 2562 patch_prober.go:28] interesting pod/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 09 20:10:38.129462 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:38.128989 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 09 20:10:39.128496 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:39.128445 2562 patch_prober.go:28] interesting pod/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 09 20:10:39.128738 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:39.128533 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 09 20:10:40.132366 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:40.132335 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:40.132858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:40.132710 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:40.133573 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:40.133539 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:10:41.573395 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:41.573363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:46.270017 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:46.269956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:51.789733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:51.789703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:51.841190 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:51.841124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:56.958952 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:56.958860 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:10:56.959526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:56.959104 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" containerID="cri-o://c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d" gracePeriod=30 Jul 09 20:10:56.972451 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:56.972420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:57.215808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:57.215720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:57.341948 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:57.341913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:57.438844 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:57.438815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:10:57.590791 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:10:57.590712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:02.229380 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.229315 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:11:02.327353 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327327 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327361 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327400 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327429 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327445 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q67b\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327565 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327626 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket\") pod \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\" (UID: \"6aa6cd8e-53e0-45fc-a744-f85b7065a40e\") " Jul 09 20:11:02.327808 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327747 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:11:02.328037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327956 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.328037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327959 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:11:02.328037 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.327973 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:11:02.328220 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.328117 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:11:02.328595 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.328554 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:11:02.329887 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.329859 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:11:02.329887 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.329871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:11:02.330033 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.329889 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b" (OuterVolumeSpecName: "kube-api-access-9q67b") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "kube-api-access-9q67b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:11:02.330033 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.329978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6aa6cd8e-53e0-45fc-a744-f85b7065a40e" (UID: "6aa6cd8e-53e0-45fc-a744-f85b7065a40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:11:02.348764 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.348731 2562 generic.go:356] "Generic (PLEG): container finished" podID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerID="c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d" exitCode=0 Jul 09 20:11:02.348898 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.348806 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" Jul 09 20:11:02.348898 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.348821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerDied","Data":"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d"} Jul 09 20:11:02.348898 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.348882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl" event={"ID":"6aa6cd8e-53e0-45fc-a744-f85b7065a40e","Type":"ContainerDied","Data":"b1d766677698dbf4f23d421bbf346ae0dfa10578f9264276b0a5d17c7d8e8695"} Jul 09 20:11:02.349023 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.348907 2562 scope.go:117] "RemoveContainer" containerID="c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d" Jul 09 20:11:02.359258 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.359242 2562 scope.go:117] "RemoveContainer" containerID="ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2" Jul 09 20:11:02.368569 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.368547 2562 scope.go:117] "RemoveContainer" containerID="c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d" Jul 09 20:11:02.368938 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:11:02.368914 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d\": container with ID starting with c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d not found: ID does not exist" containerID="c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d" Jul 09 20:11:02.369052 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.368946 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d"} err="failed to get container status \"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d\": rpc error: code = NotFound desc = could not find container \"c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d\": container with ID starting with c0ee573123049216ce093fc233ac4abad5c8ed8fef97e214ad8dbc2ac015d67d not found: ID does not exist" Jul 09 20:11:02.369052 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.368965 2562 scope.go:117] "RemoveContainer" containerID="ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2" Jul 09 20:11:02.369234 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:11:02.369216 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2\": container with ID starting with ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2 not found: ID does not exist" containerID="ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2" Jul 09 20:11:02.369277 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.369241 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2"} err="failed to get container status \"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2\": rpc error: code = NotFound desc = could not find container \"ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2\": container with ID starting with ad70e5f0695f3d6d8517400a9c74adb56a542fb1f04b1ba9b9716096738323e2 not found: ID does not exist" Jul 09 20:11:02.371517 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.371497 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:11:02.378205 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.378171 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9bad9204-openshift-default-9565778dc-pcbsl"] Jul 09 20:11:02.428964 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428928 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.428964 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428957 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.428964 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428970 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.429207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428978 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.429207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428987 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.429207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.428995 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q67b\" (UniqueName: \"kubernetes.io/projected/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-kube-api-access-9q67b\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.429207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.429005 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:02.429207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:02.429014 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa6cd8e-53e0-45fc-a744-f85b7065a40e-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:11:03.108240 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:03.108202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:03.209595 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:03.209549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:03.324831 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:03.324798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:03.488699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:03.488665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:03.953992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:03.953964 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" path="/var/lib/kubelet/pods/6aa6cd8e-53e0-45fc-a744-f85b7065a40e/volumes" Jul 09 20:11:09.091793 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:09.091755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:09.150764 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:09.150732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:16.122677 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.122638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:16.428146 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428112 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:11:16.428629 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428613 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:11:16.428629 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428630 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac27be2-1d6f-46f6-845a-fc3f4fb369ac" containerName="istio-proxy" Jul 09 20:11:16.428735 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428662 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:11:16.428735 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428669 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:11:16.428735 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428677 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:11:16.428735 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428682 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:11:16.428859 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.428758 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:11:16.431279 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.431261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.433871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.433849 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71d42f8b-openshift-default-dockercfg-mlrkk\"" Jul 09 20:11:16.444173 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.444145 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:11:16.447181 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.447158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:16.554332 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554332 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554335 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7s9v\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554596 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554596 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554596 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554812 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.554812 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.554764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655376 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655376 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s9v\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.655726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655725 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656018 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656018 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656018 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656018 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655925 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656018 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.655979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656282 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.656124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.656282 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.656208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.658527 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.658502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.658666 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.658575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.674092 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.674068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s9v\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.674250 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.674118 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token\") pod \"e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.701936 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.701852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:16.743106 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.743068 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:16.890928 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.890901 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:11:16.891813 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:11:16.891790 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f295f1_6736_4565_88cd_738603763e9a.slice/crio-654d5b2cc9834f2f8375ad6ebda6af0391638a60daedc4f61260f97641747f0f WatchSource:0}: Error finding container 654d5b2cc9834f2f8375ad6ebda6af0391638a60daedc4f61260f97641747f0f: Status 404 returned error can't find the container with id 654d5b2cc9834f2f8375ad6ebda6af0391638a60daedc4f61260f97641747f0f Jul 09 20:11:16.894409 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.894375 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:16.894511 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.894493 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:16.894598 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:16.894538 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:17.437839 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:17.437807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerStarted","Data":"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8"} Jul 09 20:11:17.438440 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:17.437860 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerStarted","Data":"654d5b2cc9834f2f8375ad6ebda6af0391638a60daedc4f61260f97641747f0f"} Jul 09 20:11:17.457671 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:17.457622 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podStartSLOduration=1.457575024 podStartE2EDuration="1.457575024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:11:17.455809994 +0000 UTC m=+1904.046570216" watchObservedRunningTime="2026-07-09 20:11:17.457575024 +0000 UTC m=+1904.048335229" Jul 09 20:11:17.743641 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:17.743530 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:17.748368 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:17.748346 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:18.443509 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:18.443476 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:18.444400 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:18.444383 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:19.399087 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:19.399039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:19.483765 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:19.483729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:19.558005 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:19.557959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:19.657086 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:19.656954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:20.063798 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:20.063651 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:21.438624 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:21.438589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:23.999490 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:23.999457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:33.445092 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.445055 2562 patch_prober.go:28] interesting pod/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 09 20:11:33.445526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.445123 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 09 20:11:33.530215 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.530178 2562 generic.go:356] "Generic (PLEG): container finished" podID="32f295f1-6736-4565-88cd-738603763e9a" containerID="244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8" exitCode=0 Jul 09 20:11:33.530419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.530255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerDied","Data":"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8"} Jul 09 20:11:33.530688 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.530673 2562 scope.go:117] "RemoveContainer" containerID="244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8" Jul 09 20:11:33.531691 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.531662 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:33.531811 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.531727 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:33.531811 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:33.531755 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:11:34.543208 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:34.543102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerStarted","Data":"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218"} Jul 09 20:11:34.744148 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:34.744114 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:34.744543 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:34.744454 2562 patch_prober.go:28] interesting pod/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 09 20:11:34.744764 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:34.744737 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 09 20:11:35.744166 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:35.744128 2562 patch_prober.go:28] interesting pod/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 09 20:11:35.744739 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:35.744703 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 09 20:11:36.326034 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:36.325998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:36.743912 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:36.743875 2562 patch_prober.go:28] interesting pod/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 09 20:11:36.744207 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:36.744179 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 09 20:11:37.744215 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:37.744173 2562 patch_prober.go:28] interesting pod/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 09 20:11:37.744852 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:37.744814 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 09 20:11:38.747862 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:38.747831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:38.748241 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:38.748111 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:38.748941 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:38.748924 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:11:53.951038 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:53.951000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:11:54.048069 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:11:54.048038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:02.594434 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.594392 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2"] Jul 09 20:12:02.595178 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.595158 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa6cd8e-53e0-45fc-a744-f85b7065a40e" containerName="istio-proxy" Jul 09 20:12:02.597885 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.597867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:02.600994 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.600662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5hwxb\"" Jul 09 20:12:02.603201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.603173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2"] Jul 09 20:12:02.693696 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.693662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2\" (UID: \"957bfd2b-b195-4b95-88bf-bcfca0b02088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:02.794238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.794199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2\" (UID: \"957bfd2b-b195-4b95-88bf-bcfca0b02088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:02.794867 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.794849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2\" (UID: \"957bfd2b-b195-4b95-88bf-bcfca0b02088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:02.910468 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:02.910434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:03.056083 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:03.056058 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2"] Jul 09 20:12:03.059208 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:12:03.059181 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957bfd2b_b195_4b95_88bf_bcfca0b02088.slice/crio-a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b WatchSource:0}: Error finding container a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b: Status 404 returned error can't find the container with id a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b Jul 09 20:12:03.715392 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:03.715354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" event={"ID":"957bfd2b-b195-4b95-88bf-bcfca0b02088","Type":"ContainerStarted","Data":"9cffc5c3dbdf3ac9c1d479cf27c58348a5ee5a945a344cd5b4fe2eb823b54c58"} Jul 09 20:12:03.715840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:03.715413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" event={"ID":"957bfd2b-b195-4b95-88bf-bcfca0b02088","Type":"ContainerStarted","Data":"a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b"} Jul 09 20:12:03.731665 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:03.731621 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" podStartSLOduration=1.731604535 podStartE2EDuration="1.731604535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:12:03.729965064 +0000 UTC m=+1950.320725284" watchObservedRunningTime="2026-07-09 20:12:03.731604535 +0000 UTC m=+1950.322364731" Jul 09 20:12:04.721945 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:04.721912 2562 generic.go:356] "Generic (PLEG): container finished" podID="957bfd2b-b195-4b95-88bf-bcfca0b02088" containerID="9cffc5c3dbdf3ac9c1d479cf27c58348a5ee5a945a344cd5b4fe2eb823b54c58" exitCode=0 Jul 09 20:12:04.722333 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:04.721973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" event={"ID":"957bfd2b-b195-4b95-88bf-bcfca0b02088","Type":"ContainerDied","Data":"9cffc5c3dbdf3ac9c1d479cf27c58348a5ee5a945a344cd5b4fe2eb823b54c58"} Jul 09 20:12:05.863392 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:05.863365 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:06.025701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.025623 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca\") pod \"957bfd2b-b195-4b95-88bf-bcfca0b02088\" (UID: \"957bfd2b-b195-4b95-88bf-bcfca0b02088\") " Jul 09 20:12:06.025962 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.025941 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "957bfd2b-b195-4b95-88bf-bcfca0b02088" (UID: "957bfd2b-b195-4b95-88bf-bcfca0b02088"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:12:06.126389 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.126361 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957bfd2b-b195-4b95-88bf-bcfca0b02088-maas-api-service-ca\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:12:06.734856 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.734824 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" event={"ID":"957bfd2b-b195-4b95-88bf-bcfca0b02088","Type":"ContainerDied","Data":"a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b"} Jul 09 20:12:06.735060 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.734869 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2" Jul 09 20:12:06.735060 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:06.734872 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a51d9afe4a4a31946caaf660936a9afe144df0dc73ebcccbb5bf7cdc0127fb2b" Jul 09 20:12:12.993094 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:12.993060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:17.677268 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:17.677230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:22.750987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:22.750947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:27.485264 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:27.485231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:34.153494 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:34.153465 2562 scope.go:117] "RemoveContainer" containerID="380a0b186f8c25dfacdfa2bee4bbaeffb5d206476a88bdef7986eee4bbea6376" Jul 09 20:12:59.704830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:59.704792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:12:59.834134 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:12:59.834096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:05.625956 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:05.625921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:17.232394 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:17.232357 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:17.288430 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:17.288401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:19.456845 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:19.456813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:19.509730 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:19.509702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:20.176341 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:20.176301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:13:20.279154 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:13:20.279116 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:14:03.637538 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:14:03.637441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:14:34.080920 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:14:34.080877 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:14:34.095728 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:14:34.095704 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:14:37.255707 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:14:37.255677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:29.809501 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:29.809462 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:16:29.810160 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:29.809737 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" containerID="cri-o://cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218" gracePeriod=30 Jul 09 20:16:29.869560 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:29.869524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:30.881103 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:30.881071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:31.286548 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:31.286507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:31.386047 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:31.386007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:31.486088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:31.486052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:32.290069 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:32.290040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:32.391475 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:32.391441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:32.488845 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:32.488771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:33.958267 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:33.958236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:35.099113 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.099092 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:16:35.100446 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100424 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100504 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100482 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7s9v\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100545 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100511 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100609 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100543 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100609 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100564 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100609 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100772 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100696 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100772 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100698 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:16:35.100772 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100776 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy\") pod \"32f295f1-6736-4565-88cd-738603763e9a\" (UID: \"32f295f1-6736-4565-88cd-738603763e9a\") " Jul 09 20:16:35.100932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100885 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:16:35.100932 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.100896 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:16:35.101088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.101025 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.101088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.101042 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.101088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.101056 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.101232 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.101204 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:16:35.101726 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.101703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:16:35.103330 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.103300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v" (OuterVolumeSpecName: "kube-api-access-v7s9v") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "kube-api-access-v7s9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:16:35.110212 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.110182 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:16:35.110364 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.110345 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:16:35.110364 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.110347 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "32f295f1-6736-4565-88cd-738603763e9a" (UID: "32f295f1-6736-4565-88cd-738603763e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:16:35.201521 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201469 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f295f1-6736-4565-88cd-738603763e9a-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.201521 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201515 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.201521 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201528 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f295f1-6736-4565-88cd-738603763e9a-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.201801 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201540 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7s9v\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-kube-api-access-v7s9v\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.201801 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201554 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f295f1-6736-4565-88cd-738603763e9a-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.201801 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.201565 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f295f1-6736-4565-88cd-738603763e9a-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:16:35.277374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.277342 2562 generic.go:356] "Generic (PLEG): container finished" podID="32f295f1-6736-4565-88cd-738603763e9a" containerID="cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218" exitCode=0 Jul 09 20:16:35.277569 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.277414 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" Jul 09 20:16:35.277569 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.277431 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerDied","Data":"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218"} Jul 09 20:16:35.277569 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.277504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8" event={"ID":"32f295f1-6736-4565-88cd-738603763e9a","Type":"ContainerDied","Data":"654d5b2cc9834f2f8375ad6ebda6af0391638a60daedc4f61260f97641747f0f"} Jul 09 20:16:35.277569 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.277532 2562 scope.go:117] "RemoveContainer" containerID="cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218" Jul 09 20:16:35.287754 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.287732 2562 scope.go:117] "RemoveContainer" containerID="244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8" Jul 09 20:16:35.296734 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.296712 2562 scope.go:117] "RemoveContainer" containerID="cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218" Jul 09 20:16:35.296988 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:16:35.296969 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218\": container with ID starting with cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218 not found: ID does not exist" containerID="cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218" Jul 09 20:16:35.297059 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.296996 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218"} err="failed to get container status \"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218\": rpc error: code = NotFound desc = could not find container \"cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218\": container with ID starting with cf4058f8e6e9f769e9fb35af854690576257adf627ccaea3e382b5638fdd5218 not found: ID does not exist" Jul 09 20:16:35.297059 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.297013 2562 scope.go:117] "RemoveContainer" containerID="244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8" Jul 09 20:16:35.297250 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:16:35.297230 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8\": container with ID starting with 244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8 not found: ID does not exist" containerID="244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8" Jul 09 20:16:35.297310 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.297257 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8"} err="failed to get container status \"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8\": rpc error: code = NotFound desc = could not find container \"244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8\": container with ID starting with 244ba2d45229e807b23579b484855d9cdcf187c8403d227f228806a6e2e230c8 not found: ID does not exist" Jul 09 20:16:35.308526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.308499 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:16:35.312355 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.312332 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71d42f8b-openshift-default-697b5797c9-29wp8"] Jul 09 20:16:35.955203 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:35.955170 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f295f1-6736-4565-88cd-738603763e9a" path="/var/lib/kubelet/pods/32f295f1-6736-4565-88cd-738603763e9a/volumes" Jul 09 20:16:39.854283 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:39.854247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:39.953789 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:39.953753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:40.570874 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:40.570838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:46.338937 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:46.338898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:46.442561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:46.442529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:46.542318 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:46.542285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:57.071100 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:57.071013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:16:57.439286 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:16:57.439250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:18.455020 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:18.454971 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:24.159265 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.159237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:24.223879 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.223847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:24.517235 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.517145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:24.805748 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.805675 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:17:24.806129 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806116 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.806171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806135 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.806171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806151 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.806171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806157 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.806171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806165 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957bfd2b-b195-4b95-88bf-bcfca0b02088" containerName="revoke-keys" Jul 09 20:17:24.806171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806171 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="957bfd2b-b195-4b95-88bf-bcfca0b02088" containerName="revoke-keys" Jul 09 20:17:24.806326 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806233 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="957bfd2b-b195-4b95-88bf-bcfca0b02088" containerName="revoke-keys" Jul 09 20:17:24.806326 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806247 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.806326 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.806253 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f295f1-6736-4565-88cd-738603763e9a" containerName="istio-proxy" Jul 09 20:17:24.809222 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.809206 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.811880 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.811861 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-35f9ad-openshift-default-dockercfg-t9xg9\"" Jul 09 20:17:24.820085 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.820064 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:17:24.825822 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.825798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:24.839753 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.839911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.839911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.839911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.840078 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.840078 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.839994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.840078 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.840023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.840223 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.840101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.840223 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.840140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940663 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.940857 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941206 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941206 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.940912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941206 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.941097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941358 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.941290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941358 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.941316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941665 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.941394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.941665 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.941522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.943147 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.943124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.943350 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.943333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.954895 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.954865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:24.955245 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:24.955229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token\") pod \"e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:25.049968 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.049933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:25.120908 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.120833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:25.270101 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.270072 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:17:25.270970 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:17:25.270942 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef363ebf_662d_45ba_bb5d_04c80f01af00.slice/crio-05a4ebb8f23f7a9bc7e7dec8d5d83d1a4de6e5349be77085a6f9037c8508fdca WatchSource:0}: Error finding container 05a4ebb8f23f7a9bc7e7dec8d5d83d1a4de6e5349be77085a6f9037c8508fdca: Status 404 returned error can't find the container with id 05a4ebb8f23f7a9bc7e7dec8d5d83d1a4de6e5349be77085a6f9037c8508fdca Jul 09 20:17:25.273184 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.273151 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:25.273262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.273218 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:25.273262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.273246 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:25.561476 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.561440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerStarted","Data":"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7"} Jul 09 20:17:25.561659 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.561497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerStarted","Data":"05a4ebb8f23f7a9bc7e7dec8d5d83d1a4de6e5349be77085a6f9037c8508fdca"} Jul 09 20:17:25.585067 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:25.585012 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podStartSLOduration=1.5849929459999998 podStartE2EDuration="1.584992946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:17:25.583167218 +0000 UTC m=+2272.173927423" watchObservedRunningTime="2026-07-09 20:17:25.584992946 +0000 UTC m=+2272.175753151" Jul 09 20:17:26.121319 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:26.121280 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:26.126396 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:26.126371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:26.567526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:26.567496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:26.568471 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:26.568452 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:27.548738 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:27.548700 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:27.665815 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:27.665787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:27.768531 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:27.768497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:27.866459 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:27.866371 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:28.208702 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:28.208669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:28.305934 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:28.305905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:32.102548 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.102518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:32.472541 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.472503 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:32.759021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.758943 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:17:32.762710 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.762688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.765817 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.765796 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-54be63-openshift-default-dockercfg-tvs2w\"" Jul 09 20:17:32.774338 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.774310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:17:32.789025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.789001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:32.797504 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797599 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484zj\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797650 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797650 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797729 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797729 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.797809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.797784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.898980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.898943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.898980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.898981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899039 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484zj\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899604 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899604 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899604 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.899740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.899727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.901787 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.901762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.901970 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.901951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.907308 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.907285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:32.907484 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:32.907462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-484zj\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj\") pod \"e2e-inf-b-54be63-openshift-default-56654d67c-hh56b\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:33.050031 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.049954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:33.076143 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.076117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:33.220502 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.220472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:17:33.222374 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:17:33.222347 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f26eb8b_3e58_47bd_9533_5a551b64db38.slice/crio-7779e361c0f8d88120aa6fb6306f31328a64c49e8ee30f64407e3756198ee542 WatchSource:0}: Error finding container 7779e361c0f8d88120aa6fb6306f31328a64c49e8ee30f64407e3756198ee542: Status 404 returned error can't find the container with id 7779e361c0f8d88120aa6fb6306f31328a64c49e8ee30f64407e3756198ee542 Jul 09 20:17:33.224624 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.224559 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:33.224724 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.224659 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:33.224724 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.224688 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:33.611479 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.611437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerStarted","Data":"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165"} Jul 09 20:17:33.611703 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.611504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerStarted","Data":"7779e361c0f8d88120aa6fb6306f31328a64c49e8ee30f64407e3756198ee542"} Jul 09 20:17:33.653068 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:33.653018 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podStartSLOduration=1.653002863 podStartE2EDuration="1.653002863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:17:33.649574773 +0000 UTC m=+2280.240334999" watchObservedRunningTime="2026-07-09 20:17:33.653002863 +0000 UTC m=+2280.243763067" Jul 09 20:17:34.076435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:34.076397 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:34.081131 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:34.081094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:34.617372 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:34.617333 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:34.618549 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:34.618528 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:35.636255 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:35.636222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:35.717283 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:35.717245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:35.804075 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:35.804045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:35.903562 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:35.903524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:36.284059 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:36.283934 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:36.389864 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:36.389828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:40.202665 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:40.202628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:42.609806 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:42.609758 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 20:17:42.610286 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:42.609856 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 20:17:44.688466 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.688434 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerID="4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7" exitCode=0 Jul 09 20:17:44.688890 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.688513 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerDied","Data":"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7"} Jul 09 20:17:44.688999 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.688984 2562 scope.go:117] "RemoveContainer" containerID="4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7" Jul 09 20:17:44.690066 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.690039 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:44.690163 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.690101 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:44.690163 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:44.690135 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:45.702236 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:45.702201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerStarted","Data":"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562"} Jul 09 20:17:46.121837 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:46.121744 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:46.122145 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:46.122123 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 09 20:17:46.122293 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:46.122274 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 09 20:17:47.121395 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:47.121362 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 09 20:17:47.121827 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:47.121421 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 09 20:17:48.122226 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:48.122142 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 09 20:17:48.122226 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:48.122200 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 09 20:17:48.967591 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:48.967539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:49.065618 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:49.065564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:49.121988 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:49.121950 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 09 20:17:49.122193 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:49.122016 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 09 20:17:50.126006 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:50.125976 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:50.126503 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:50.126381 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:50.127170 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:50.127150 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:17:50.658807 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:50.658763 2562 patch_prober.go:28] interesting pod/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 20:17:50.659021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:50.658850 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 20:17:51.740673 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.740634 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerID="08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165" exitCode=0 Jul 09 20:17:51.741097 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.740714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerDied","Data":"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165"} Jul 09 20:17:51.741222 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.741203 2562 scope.go:117] "RemoveContainer" containerID="08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165" Jul 09 20:17:51.742152 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.742128 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:51.742210 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.742190 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:51.742248 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:51.742222 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:17:52.376392 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:52.376360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:52.752648 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:52.752615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerStarted","Data":"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a"} Jul 09 20:17:53.076518 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:53.076429 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:53.076699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:53.076663 2562 patch_prober.go:28] interesting pod/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 09 20:17:53.076744 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:53.076717 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 09 20:17:54.077133 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:54.077089 2562 patch_prober.go:28] interesting pod/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 09 20:17:54.077544 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:54.077168 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 09 20:17:55.077544 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:55.077505 2562 patch_prober.go:28] interesting pod/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 09 20:17:55.078072 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:55.077565 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 09 20:17:56.076873 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:56.076835 2562 patch_prober.go:28] interesting pod/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 09 20:17:56.077141 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:56.076890 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 09 20:17:57.079821 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:57.079788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:57.080236 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:57.080066 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:57.080827 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:57.080807 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:17:57.969858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:57.969827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:17:58.071213 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:17:58.071184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:18:02.497798 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:18:02.497765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:18:20.904348 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:18:20.904312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:18:21.001591 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:18:21.001543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:18:31.300623 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:18:31.300516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:04.590842 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:04.590808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:05.508148 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:05.508113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:06.257796 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:06.257765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:06.359198 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:06.359167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:16.115433 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:16.115392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:20.744801 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:20.744769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:26.424405 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:26.424363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:26.582981 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:26.582932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:31.501109 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:31.501072 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:19:31.501541 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:31.501328 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" containerID="cri-o://fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562" gracePeriod=30 Jul 09 20:19:31.517873 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:31.517843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:31.775880 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:31.775768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:32.655701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:32.655659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:32.756779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:32.756748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:34.122232 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:34.122200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:19:34.139372 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:34.139348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:19:35.128957 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:35.128922 2562 patch_prober.go:28] interesting pod/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 20:19:35.129366 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:35.128980 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 20:19:36.764031 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.763974 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:19:36.850499 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850464 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850556 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850602 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850639 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850867 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850807 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850867 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850972 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.850972 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.851077 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.850973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket\") pod \"ef363ebf-662d-45ba-bb5d-04c80f01af00\" (UID: \"ef363ebf-662d-45ba-bb5d-04c80f01af00\") " Jul 09 20:19:36.851129 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851108 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:19:36.851332 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851285 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:19:36.851332 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851292 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:19:36.851778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851368 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.851778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851464 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:19:36.851778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.851701 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:19:36.853020 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.852998 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:19:36.853182 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.853162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:19:36.853471 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.853443 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:19:36.853471 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.853450 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn" (OuterVolumeSpecName: "kube-api-access-4v9cn") pod "ef363ebf-662d-45ba-bb5d-04c80f01af00" (UID: "ef363ebf-662d-45ba-bb5d-04c80f01af00"). InnerVolumeSpecName "kube-api-access-4v9cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:19:36.951989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.951958 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.951989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.951983 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.951989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.951992 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-kube-api-access-4v9cn\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.952214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.952003 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.952214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.952011 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef363ebf-662d-45ba-bb5d-04c80f01af00-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.952214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.952019 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef363ebf-662d-45ba-bb5d-04c80f01af00-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.952214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.952026 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:36.952214 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:36.952034 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef363ebf-662d-45ba-bb5d-04c80f01af00-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:19:37.363547 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.363511 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerID="fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562" exitCode=0 Jul 09 20:19:37.363746 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.363570 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" Jul 09 20:19:37.363746 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.363619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerDied","Data":"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562"} Jul 09 20:19:37.363746 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.363681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb" event={"ID":"ef363ebf-662d-45ba-bb5d-04c80f01af00","Type":"ContainerDied","Data":"05a4ebb8f23f7a9bc7e7dec8d5d83d1a4de6e5349be77085a6f9037c8508fdca"} Jul 09 20:19:37.363746 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.363707 2562 scope.go:117] "RemoveContainer" containerID="fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562" Jul 09 20:19:37.374526 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.374510 2562 scope.go:117] "RemoveContainer" containerID="4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7" Jul 09 20:19:37.383268 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.383249 2562 scope.go:117] "RemoveContainer" containerID="fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562" Jul 09 20:19:37.383605 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:19:37.383548 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562\": container with ID starting with fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562 not found: ID does not exist" containerID="fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562" Jul 09 20:19:37.383682 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.383612 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562"} err="failed to get container status \"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562\": rpc error: code = NotFound desc = could not find container \"fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562\": container with ID starting with fc40de1c1dd004503a3d5eaa686fd8ceaef2b96bd32ebab2e763fd84ea576562 not found: ID does not exist" Jul 09 20:19:37.383682 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.383671 2562 scope.go:117] "RemoveContainer" containerID="4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7" Jul 09 20:19:37.383973 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:19:37.383951 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7\": container with ID starting with 4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7 not found: ID does not exist" containerID="4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7" Jul 09 20:19:37.384079 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.383979 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7"} err="failed to get container status \"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7\": rpc error: code = NotFound desc = could not find container \"4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7\": container with ID starting with 4f9a0a4508c1cbc710b6377daf9461a035ae4e3f24d4fbd49890ea81216cb8b7 not found: ID does not exist" Jul 09 20:19:37.386649 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.386631 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:19:37.391398 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.391378 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-35f9ad-openshift-default-68f7b856fc-4mgvb"] Jul 09 20:19:37.953182 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:37.953144 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" path="/var/lib/kubelet/pods/ef363ebf-662d-45ba-bb5d-04c80f01af00/volumes" Jul 09 20:19:42.329071 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:42.329038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:46.989593 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:46.989549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:52.676507 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:52.676477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:52.735075 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:52.735042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:55.436265 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:55.436185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:57.667429 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:57.667393 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:19:57.667868 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:57.667682 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" containerID="cri-o://a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a" gracePeriod=30 Jul 09 20:19:57.709209 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:57.709174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:57.955563 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:57.955477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:58.347206 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.347122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:58.686192 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.686159 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:19:58.686601 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.686560 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:19:58.686601 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.686572 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:19:58.686708 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.686663 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:19:58.686708 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.686672 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:19:58.688984 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.688966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.692599 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.692558 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b7d55f-openshift-default-dockercfg-64c8d\"" Jul 09 20:19:58.713811 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.713782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:58.733732 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.733704 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:19:58.743140 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmtq\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743532 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743532 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743532 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.743532 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.743472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844306 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844306 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmtq\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844466 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844617 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.844989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.845100 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.844983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.845100 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.845059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.845208 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.845162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.846762 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.846745 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.847297 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.847275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.854840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.854819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmtq\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.861754 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.861732 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token\") pod \"e2e-api-a-b7d55f-openshift-default-857997c665-mhztp\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:58.999868 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:58.999771 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:19:59.149274 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.149248 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:19:59.150775 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:19:59.150747 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d0d79e_78c7_485d_8629_c2a2db5055fb.slice/crio-e36980e273cb070735b614947099fd3caefb32bf68bc6e3eec59e6111c40b3dd WatchSource:0}: Error finding container e36980e273cb070735b614947099fd3caefb32bf68bc6e3eec59e6111c40b3dd: Status 404 returned error can't find the container with id e36980e273cb070735b614947099fd3caefb32bf68bc6e3eec59e6111c40b3dd Jul 09 20:19:59.152855 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.152822 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:19:59.152951 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.152902 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:19:59.152951 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.152946 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:19:59.366538 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.366442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:19:59.494068 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.494027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerStarted","Data":"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10"} Jul 09 20:19:59.494271 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.494092 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerStarted","Data":"e36980e273cb070735b614947099fd3caefb32bf68bc6e3eec59e6111c40b3dd"} Jul 09 20:19:59.518969 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:19:59.518924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podStartSLOduration=1.5189080050000001 podStartE2EDuration="1.518908005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:19:59.517995202 +0000 UTC m=+2426.108755424" watchObservedRunningTime="2026-07-09 20:19:59.518908005 +0000 UTC m=+2426.109668210" Jul 09 20:20:00.000596 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:00.000552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:00.005481 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:00.005459 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:00.500467 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:00.500434 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:00.501473 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:00.501458 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:01.478185 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:01.478154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:01.584686 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:01.584655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:01.838144 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:01.838045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:02.177121 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.177078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:02.279597 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.279552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:02.931425 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.931356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:20:02.975685 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975742 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975787 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975824 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-484zj\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.975871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.975846 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.976169 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976007 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.976169 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976039 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:02.976169 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976078 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy\") pod \"2f26eb8b-3e58-47bd-9533-5a551b64db38\" (UID: \"2f26eb8b-3e58-47bd-9533-5a551b64db38\") " Jul 09 20:20:02.976322 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976295 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:02.976377 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:20:02.976441 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976420 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:02.976480 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976444 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:02.976480 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976462 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f26eb8b-3e58-47bd-9533-5a551b64db38-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:02.976557 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:02.976666 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.976638 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:02.978699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.978669 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:02.978831 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.978728 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:20:02.978973 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.978944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:20:02.979217 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:02.979192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj" (OuterVolumeSpecName: "kube-api-access-484zj") pod "2f26eb8b-3e58-47bd-9533-5a551b64db38" (UID: "2f26eb8b-3e58-47bd-9533-5a551b64db38"). InnerVolumeSpecName "kube-api-access-484zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077663 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484zj\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-kube-api-access-484zj\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077704 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077718 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077731 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077743 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.077779 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.077755 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f26eb8b-3e58-47bd-9533-5a551b64db38-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:03.518661 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.518627 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerID="a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a" exitCode=0 Jul 09 20:20:03.518840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.518682 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" Jul 09 20:20:03.518840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.518718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerDied","Data":"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a"} Jul 09 20:20:03.518840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.518780 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b" event={"ID":"2f26eb8b-3e58-47bd-9533-5a551b64db38","Type":"ContainerDied","Data":"7779e361c0f8d88120aa6fb6306f31328a64c49e8ee30f64407e3756198ee542"} Jul 09 20:20:03.518840 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.518804 2562 scope.go:117] "RemoveContainer" containerID="a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a" Jul 09 20:20:03.529010 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.528994 2562 scope.go:117] "RemoveContainer" containerID="08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165" Jul 09 20:20:03.537518 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.537498 2562 scope.go:117] "RemoveContainer" containerID="a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a" Jul 09 20:20:03.537792 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:20:03.537771 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a\": container with ID starting with a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a not found: ID does not exist" containerID="a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a" Jul 09 20:20:03.537849 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.537801 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a"} err="failed to get container status \"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a\": rpc error: code = NotFound desc = could not find container \"a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a\": container with ID starting with a8ac3e59848a55e9ae6d140cc85e533e8f0e4d6ec90e825705b18d892410f41a not found: ID does not exist" Jul 09 20:20:03.537849 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.537820 2562 scope.go:117] "RemoveContainer" containerID="08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165" Jul 09 20:20:03.538084 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:20:03.538069 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165\": container with ID starting with 08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165 not found: ID does not exist" containerID="08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165" Jul 09 20:20:03.538124 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.538089 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165"} err="failed to get container status \"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165\": rpc error: code = NotFound desc = could not find container \"08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165\": container with ID starting with 08c99fdea87bb24be3d967e965691ff47c4505c5e52273b576efd402e7c8c165 not found: ID does not exist" Jul 09 20:20:03.542438 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.542415 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:20:03.546341 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.546318 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-54be63-openshift-default-56654d67c-hh56b"] Jul 09 20:20:03.954558 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:03.954518 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" path="/var/lib/kubelet/pods/2f26eb8b-3e58-47bd-9533-5a551b64db38/volumes" Jul 09 20:20:06.015831 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.015786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:06.515115 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.515079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:06.794750 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.794664 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:20:06.795146 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795128 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:20:06.795238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795150 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef363ebf-662d-45ba-bb5d-04c80f01af00" containerName="istio-proxy" Jul 09 20:20:06.795238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795167 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.795238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795175 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.795238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795196 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.795238 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795204 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.795699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795309 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.795699 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.795322 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f26eb8b-3e58-47bd-9533-5a551b64db38" containerName="istio-proxy" Jul 09 20:20:06.799137 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.799115 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.801894 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.801873 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e062b5-openshift-default-dockercfg-27jj2\"" Jul 09 20:20:06.809555 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809668 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:20:06.809668 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809758 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn9c\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809800 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.809955 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.810007 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.810007 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.809971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.824725 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.824695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:06.911087 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911093 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn9c\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911356 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911817 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.911817 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.911762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.912072 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.912050 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.912171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.912052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.912171 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.912095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.912261 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.912216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.914148 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.914129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.914309 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.914293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.926246 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.926222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn9c\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:06.930732 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:06.930713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token\") pod \"e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:07.111156 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.111068 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:07.259343 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.259314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:20:07.260713 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:20:07.260689 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364786f7_af28_4049_9d81_9cf67fc2b347.slice/crio-53d5263687e6e990d079cbafe7ee70c3f67bc239ca7711e149eec8d79efa792c WatchSource:0}: Error finding container 53d5263687e6e990d079cbafe7ee70c3f67bc239ca7711e149eec8d79efa792c: Status 404 returned error can't find the container with id 53d5263687e6e990d079cbafe7ee70c3f67bc239ca7711e149eec8d79efa792c Jul 09 20:20:07.262904 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.262873 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:07.262987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.262936 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:07.262987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.262962 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:07.398446 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.398418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:07.545938 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.545896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerStarted","Data":"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19"} Jul 09 20:20:07.546107 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.545958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerStarted","Data":"53d5263687e6e990d079cbafe7ee70c3f67bc239ca7711e149eec8d79efa792c"} Jul 09 20:20:07.567193 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:07.567134 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podStartSLOduration=1.5671158269999999 podStartE2EDuration="1.567115827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:20:07.566515554 +0000 UTC m=+2434.157275788" watchObservedRunningTime="2026-07-09 20:20:07.567115827 +0000 UTC m=+2434.157876035" Jul 09 20:20:08.112221 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:08.112186 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:08.117870 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:08.117846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:08.551986 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:08.551957 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:08.553002 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:08.552983 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:09.551239 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:09.551201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:09.643239 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:09.643186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:09.837633 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:09.837475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:10.031335 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:10.031274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:10.384733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:10.384674 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:10.485175 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:10.485136 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:14.070641 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:14.070541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:16.541782 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:16.541739 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 20:20:16.542162 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:16.541803 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 20:20:17.612684 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.612649 2562 generic.go:356] "Generic (PLEG): container finished" podID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerID="8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10" exitCode=0 Jul 09 20:20:17.613217 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.612728 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerDied","Data":"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10"} Jul 09 20:20:17.613305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.613286 2562 scope.go:117] "RemoveContainer" containerID="8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10" Jul 09 20:20:17.614305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.614281 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:17.614409 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.614360 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:17.614409 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:17.614388 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:18.620369 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:18.620331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerStarted","Data":"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c"} Jul 09 20:20:19.000662 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:19.000623 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:19.000929 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:19.000905 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 09 20:20:19.001025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:19.000971 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 09 20:20:20.000628 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:20.000561 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 09 20:20:20.001004 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:20.000643 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 09 20:20:21.000931 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:21.000883 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 09 20:20:21.001428 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:21.000970 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 09 20:20:22.000823 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:22.000782 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 09 20:20:22.001035 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:22.000851 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 09 20:20:22.815051 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:22.815017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:22.915472 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:22.915440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:23.004523 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:23.004484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:23.004977 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:23.004785 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:23.005679 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:23.005664 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:24.011444 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.010740 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:58444->10.133.0.69:15021: read: connection reset by peer" start-of-body= Jul 09 20:20:24.011444 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.010820 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:58444->10.133.0.69:15021: read: connection reset by peer" Jul 09 20:20:24.664716 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.664683 2562 generic.go:356] "Generic (PLEG): container finished" podID="364786f7-af28-4049-9d81-9cf67fc2b347" containerID="05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19" exitCode=0 Jul 09 20:20:24.664926 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.664759 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerDied","Data":"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19"} Jul 09 20:20:24.665271 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.665248 2562 scope.go:117] "RemoveContainer" containerID="05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19" Jul 09 20:20:24.666415 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.666385 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:24.666568 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.666468 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:24.666568 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:24.666529 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:20:25.673429 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:25.673396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerStarted","Data":"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b"} Jul 09 20:20:26.111627 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:26.111512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:26.111831 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:26.111807 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 09 20:20:26.111888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:26.111867 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 09 20:20:27.112405 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:27.112370 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 09 20:20:27.112847 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:27.112424 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 09 20:20:28.112158 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:28.112119 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 09 20:20:28.112337 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:28.112183 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 09 20:20:29.112264 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:29.112231 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 09 20:20:29.112674 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:29.112284 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 09 20:20:30.114985 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:30.114958 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:30.115350 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:30.115318 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:30.115893 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:30.115880 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:20:32.086443 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:32.086404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:32.189855 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:32.189815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:32.814361 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:32.814327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:37.396693 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:37.396643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:43.297952 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:43.297918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:43.368201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:43.368168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:48.040393 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:48.040345 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:20:48.040888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:48.040634 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" containerID="cri-o://2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c" gracePeriod=30 Jul 09 20:20:48.055780 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:48.055754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:48.300098 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:48.299995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:20:53.007710 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.007662 2562 patch_prober.go:28] interesting pod/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 20:20:53.008154 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.007740 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 20:20:53.296958 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.296932 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:53.358511 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358475 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358542 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwmtq\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358676 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358833 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358833 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358774 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358833 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358829 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358858 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.358987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358894 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data\") pod \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\" (UID: \"b6d0d79e-78c7-485d-8629-c2a2db5055fb\") " Jul 09 20:20:53.359098 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.358964 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:53.359159 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359123 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:20:53.359387 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359306 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:53.359509 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359476 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:53.359778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359520 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.359778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359542 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.359778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.359556 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.360111 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.360085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:53.361019 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.360999 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:20:53.361167 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.361144 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:20:53.361288 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.361269 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:20:53.361288 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.361275 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq" (OuterVolumeSpecName: "kube-api-access-dwmtq") pod "b6d0d79e-78c7-485d-8629-c2a2db5055fb" (UID: "b6d0d79e-78c7-485d-8629-c2a2db5055fb"). InnerVolumeSpecName "kube-api-access-dwmtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:20:53.460133 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460082 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.460133 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460125 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.460133 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460137 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.460387 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460150 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmtq\" (UniqueName: \"kubernetes.io/projected/b6d0d79e-78c7-485d-8629-c2a2db5055fb-kube-api-access-dwmtq\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.460387 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460162 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.460387 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.460174 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d0d79e-78c7-485d-8629-c2a2db5055fb-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:20:53.832543 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.832506 2562 generic.go:356] "Generic (PLEG): container finished" podID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerID="2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c" exitCode=0 Jul 09 20:20:53.832769 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.832570 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" Jul 09 20:20:53.832769 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.832602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerDied","Data":"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c"} Jul 09 20:20:53.832769 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.832656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp" event={"ID":"b6d0d79e-78c7-485d-8629-c2a2db5055fb","Type":"ContainerDied","Data":"e36980e273cb070735b614947099fd3caefb32bf68bc6e3eec59e6111c40b3dd"} Jul 09 20:20:53.832769 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.832678 2562 scope.go:117] "RemoveContainer" containerID="2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c" Jul 09 20:20:53.843384 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.843367 2562 scope.go:117] "RemoveContainer" containerID="8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10" Jul 09 20:20:53.852201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.852177 2562 scope.go:117] "RemoveContainer" containerID="2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c" Jul 09 20:20:53.852458 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:20:53.852438 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c\": container with ID starting with 2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c not found: ID does not exist" containerID="2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c" Jul 09 20:20:53.852514 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.852467 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c"} err="failed to get container status \"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c\": rpc error: code = NotFound desc = could not find container \"2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c\": container with ID starting with 2cec78891c88d5ea9f9b710fbc504d611fbbcc5cae9a937ac7a3063b0ce4240c not found: ID does not exist" Jul 09 20:20:53.852514 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.852486 2562 scope.go:117] "RemoveContainer" containerID="8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10" Jul 09 20:20:53.852773 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:20:53.852754 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10\": container with ID starting with 8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10 not found: ID does not exist" containerID="8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10" Jul 09 20:20:53.852852 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.852786 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10"} err="failed to get container status \"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10\": rpc error: code = NotFound desc = could not find container \"8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10\": container with ID starting with 8fdb86b67e2eb65818781d4f9fc212fe79a6b23be3cbe3f0150d99f8210f0d10 not found: ID does not exist" Jul 09 20:20:53.858619 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.858566 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:20:53.866972 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.866948 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b7d55f-openshift-default-857997c665-mhztp"] Jul 09 20:20:53.953513 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:53.953480 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" path="/var/lib/kubelet/pods/b6d0d79e-78c7-485d-8629-c2a2db5055fb/volumes" Jul 09 20:20:59.119463 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:20:59.119431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:03.573753 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:03.573712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:09.772854 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:09.772820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:09.874411 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:09.874376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:14.264045 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:14.263997 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:21:14.264508 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:14.264408 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" containerID="cri-o://8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b" gracePeriod=30 Jul 09 20:21:14.282039 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:14.282000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:14.519455 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:14.519333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:14.871888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:14.871802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:15.118194 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.118153 2562 patch_prober.go:28] interesting pod/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 20:21:15.118363 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.118240 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 20:21:15.204572 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.204536 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:21:15.204972 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.204959 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.205025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.204974 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.205025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.204988 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.205025 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.204994 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.205140 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.205058 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.205140 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.205067 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d0d79e-78c7-485d-8629-c2a2db5055fb" containerName="istio-proxy" Jul 09 20:21:15.207439 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.207423 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.210952 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.210933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7b406b-openshift-default-dockercfg-f5rnb\"" Jul 09 20:21:15.221160 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.221133 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:21:15.229604 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.229565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:15.362790 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362834 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfknv\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.362981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.363011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.363031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.363315 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.363163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.464785 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.464785 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465021 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.464904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfknv\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465303 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465303 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465303 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465545 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465822 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.465956 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.465935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.466073 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.466044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.466342 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.466319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.468267 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.468242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:15.468739 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.468716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.469179 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.469161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.475069 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.475043 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.475266 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.475247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfknv\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv\") pod \"e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.517635 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.517603 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:15.655901 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.655876 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:21:15.658030 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:21:15.658000 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36570b9_7c85_4ea5_8ba8_2c961da56aec.slice/crio-73cf9aec29270e585bebf4502900ab0011d0336a4da63c52d014edd047466db0 WatchSource:0}: Error finding container 73cf9aec29270e585bebf4502900ab0011d0336a4da63c52d014edd047466db0: Status 404 returned error can't find the container with id 73cf9aec29270e585bebf4502900ab0011d0336a4da63c52d014edd047466db0 Jul 09 20:21:15.660209 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.660178 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:15.660291 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.660239 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:15.660291 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.660268 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:15.963659 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.963625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerStarted","Data":"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5"} Jul 09 20:21:15.963860 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.963675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerStarted","Data":"73cf9aec29270e585bebf4502900ab0011d0336a4da63c52d014edd047466db0"} Jul 09 20:21:15.993294 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:15.993205 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podStartSLOduration=0.993190004 podStartE2EDuration="993.190004ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:21:15.990905841 +0000 UTC m=+2502.581666046" watchObservedRunningTime="2026-07-09 20:21:15.993190004 +0000 UTC m=+2502.583950255" Jul 09 20:21:16.518684 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:16.518651 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:16.523331 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:16.523309 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:16.970316 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:16.970288 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:16.971367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:16.971347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:17.932466 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:17.932439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:18.007965 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:18.007930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:18.094039 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:18.094003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:18.190655 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:18.190560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:18.518936 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:18.518854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:18.619423 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:18.619391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:19.521659 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.521637 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:21:19.603855 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.603778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.603855 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.603838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604058 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.603888 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604058 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.603919 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604058 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.603991 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604058 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604014 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604058 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604054 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqn9c\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604077 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604115 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data\") pod \"364786f7-af28-4049-9d81-9cf67fc2b347\" (UID: \"364786f7-af28-4049-9d81-9cf67fc2b347\") " Jul 09 20:21:19.604305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604211 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:21:19.604305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604198 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:21:19.604305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:21:19.604567 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604403 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:21:19.604651 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604638 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.604704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604659 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.604704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604673 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364786f7-af28-4049-9d81-9cf67fc2b347-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.604704 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.604688 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.605659 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.605633 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data" (OuterVolumeSpecName: "istio-data") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:21:19.606412 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.606386 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:21:19.606750 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.606730 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token" (OuterVolumeSpecName: "istio-token") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:21:19.606907 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.606880 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:21:19.606907 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.606893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c" (OuterVolumeSpecName: "kube-api-access-xqn9c") pod "364786f7-af28-4049-9d81-9cf67fc2b347" (UID: "364786f7-af28-4049-9d81-9cf67fc2b347"). InnerVolumeSpecName "kube-api-access-xqn9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:21:19.705385 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.705354 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364786f7-af28-4049-9d81-9cf67fc2b347-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.705385 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.705382 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.705553 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.705393 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqn9c\" (UniqueName: \"kubernetes.io/projected/364786f7-af28-4049-9d81-9cf67fc2b347-kube-api-access-xqn9c\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.705553 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.705410 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.705553 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.705419 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364786f7-af28-4049-9d81-9cf67fc2b347-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:21:19.990012 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.989981 2562 generic.go:356] "Generic (PLEG): container finished" podID="364786f7-af28-4049-9d81-9cf67fc2b347" containerID="8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b" exitCode=0 Jul 09 20:21:19.990157 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.990085 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" Jul 09 20:21:19.990157 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.990116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerDied","Data":"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b"} Jul 09 20:21:19.990249 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.990164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz" event={"ID":"364786f7-af28-4049-9d81-9cf67fc2b347","Type":"ContainerDied","Data":"53d5263687e6e990d079cbafe7ee70c3f67bc239ca7711e149eec8d79efa792c"} Jul 09 20:21:19.990249 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:19.990185 2562 scope.go:117] "RemoveContainer" containerID="8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b" Jul 09 20:21:20.001724 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.001666 2562 scope.go:117] "RemoveContainer" containerID="05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19" Jul 09 20:21:20.010431 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.010413 2562 scope.go:117] "RemoveContainer" containerID="8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b" Jul 09 20:21:20.010538 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.010518 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:21:20.010733 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:21:20.010711 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b\": container with ID starting with 8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b not found: ID does not exist" containerID="8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b" Jul 09 20:21:20.010794 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.010742 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b"} err="failed to get container status \"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b\": rpc error: code = NotFound desc = could not find container \"8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b\": container with ID starting with 8dd3a5e0d85b1d491a32c5edc58a1a0c9d58eb2e71515e4ffc598e47622b803b not found: ID does not exist" Jul 09 20:21:20.010794 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.010759 2562 scope.go:117] "RemoveContainer" containerID="05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19" Jul 09 20:21:20.010974 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:21:20.010951 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19\": container with ID starting with 05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19 not found: ID does not exist" containerID="05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19" Jul 09 20:21:20.011036 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.010983 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19"} err="failed to get container status \"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19\": rpc error: code = NotFound desc = could not find container \"05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19\": container with ID starting with 05ea85c965c4510b35954019013d0b35b3a59509cc90a0759cb58833dc3c1c19 not found: ID does not exist" Jul 09 20:21:20.016036 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:20.016014 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e062b5-openshift-default-79976c6989-pnzmz"] Jul 09 20:21:21.954467 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:21.954433 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" path="/var/lib/kubelet/pods/364786f7-af28-4049-9d81-9cf67fc2b347/volumes" Jul 09 20:21:22.501756 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:22.501719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:22.844830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:22.844748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:23.150818 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.150785 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:21:23.151277 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.151209 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:21:23.151277 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.151222 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:21:23.151382 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.151284 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:21:23.154727 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.154707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.157787 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.157770 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-3452a7-openshift-default-dockercfg-fhb74\"" Jul 09 20:21:23.165811 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.165783 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:21:23.180162 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.180127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:23.339419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339626 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339626 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339626 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339626 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339803 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339803 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339803 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kw4\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.339803 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.339754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kw4\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.440992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.440984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441220 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441220 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441220 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441450 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441540 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441613 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.441613 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.441570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.443455 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.443428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.443565 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.443536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.450535 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.450505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.450888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.450867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kw4\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4\") pod \"e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.466717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.466694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:23.468117 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.468093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:23.610314 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.610281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:21:23.611642 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:21:23.611618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5553a71a_cd5a_4d13_a4f1_486b264c8f31.slice/crio-6fdf56ff8c4981ca4246fef8b0f1854939871774d4fed02728b5ec44d1a521b4 WatchSource:0}: Error finding container 6fdf56ff8c4981ca4246fef8b0f1854939871774d4fed02728b5ec44d1a521b4: Status 404 returned error can't find the container with id 6fdf56ff8c4981ca4246fef8b0f1854939871774d4fed02728b5ec44d1a521b4 Jul 09 20:21:23.613985 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.613944 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:23.614073 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.614041 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:23.614113 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:23.614090 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:24.016319 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:24.016286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerStarted","Data":"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df"} Jul 09 20:21:24.016543 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:24.016339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerStarted","Data":"6fdf56ff8c4981ca4246fef8b0f1854939871774d4fed02728b5ec44d1a521b4"} Jul 09 20:21:24.043717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:24.043664 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podStartSLOduration=1.043644635 podStartE2EDuration="1.043644635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:21:24.043129642 +0000 UTC m=+2510.633889874" watchObservedRunningTime="2026-07-09 20:21:24.043644635 +0000 UTC m=+2510.634404841" Jul 09 20:21:24.467868 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:24.467840 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:24.472727 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:24.472704 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:25.021828 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:25.021801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:25.022838 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:25.022820 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:26.048640 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.048603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:26.141743 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.141709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:26.237443 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.237410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:26.375916 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.375841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:26.824557 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.824523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:26.925278 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:26.925247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:30.555799 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:30.555762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:33.012846 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:33.012805 2562 patch_prober.go:28] interesting pod/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 20:21:33.013264 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:33.012893 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 20:21:34.089966 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.089934 2562 generic.go:356] "Generic (PLEG): container finished" podID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerID="80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5" exitCode=0 Jul 09 20:21:34.090360 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.090006 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerDied","Data":"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5"} Jul 09 20:21:34.090446 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.090428 2562 scope.go:117] "RemoveContainer" containerID="80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5" Jul 09 20:21:34.091491 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.091464 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:34.091640 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.091532 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:34.091640 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:34.091566 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:35.097509 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:35.097477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerStarted","Data":"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7"} Jul 09 20:21:35.518571 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:35.518536 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:35.518781 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:35.518675 2562 patch_prober.go:28] interesting pod/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 09 20:21:35.518781 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:35.518710 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 09 20:21:36.518888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:36.518849 2562 patch_prober.go:28] interesting pod/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 09 20:21:36.519274 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:36.518905 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 09 20:21:37.518469 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:37.518433 2562 patch_prober.go:28] interesting pod/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 09 20:21:37.518664 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:37.518519 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 09 20:21:38.519074 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:38.519032 2562 patch_prober.go:28] interesting pod/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 09 20:21:38.519463 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:38.519098 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 09 20:21:39.521528 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:39.521500 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:39.521952 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:39.521860 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:39.522494 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:39.522476 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:21:39.735995 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:39.735959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:40.378981 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:40.378173 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": read tcp 10.133.0.2:46256->10.133.0.71:15021: read: connection reset by peer" start-of-body= Jul 09 20:21:40.378981 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:40.378234 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": read tcp 10.133.0.2:46256->10.133.0.71:15021: read: connection reset by peer" Jul 09 20:21:41.139770 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.139738 2562 generic.go:356] "Generic (PLEG): container finished" podID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerID="68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df" exitCode=0 Jul 09 20:21:41.140145 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.139811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerDied","Data":"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df"} Jul 09 20:21:41.140305 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.140292 2562 scope.go:117] "RemoveContainer" containerID="68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df" Jul 09 20:21:41.141299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.141272 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:41.141373 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.141356 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:41.141423 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.141407 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 09 20:21:41.259262 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:41.259225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:42.152360 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:42.152311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerStarted","Data":"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac"} Jul 09 20:21:42.467566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:42.467474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:42.467751 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:42.467697 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 09 20:21:42.467751 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:42.467742 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 09 20:21:43.467871 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:43.467834 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 09 20:21:43.468252 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:43.467904 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 09 20:21:44.467401 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:44.467366 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 09 20:21:44.467604 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:44.467433 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 09 20:21:45.467170 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:45.467129 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 09 20:21:45.467695 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:45.467196 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 09 20:21:46.470621 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:46.470592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:46.471105 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:46.470842 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:46.471503 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:46.471489 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:21:47.592436 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:47.592400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:47.679911 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:47.679873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:21:47.749441 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:21:47.749405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:22:13.741953 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:22:13.741923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:23:01.828914 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:23:01.828825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:23:19.361968 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:23:19.361935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:23:21.640992 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:23:21.640944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:23:21.738434 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:23:21.738401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:24:25.098936 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:24:25.098856 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:24:26.032005 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:24:26.031968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:24:31.905417 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:24:31.905376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:24:34.163434 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:24:34.163406 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:24:34.178376 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:24:34.178355 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:25:14.091229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:25:14.091196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:25:54.608661 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:25:54.608560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:25:54.988889 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:25:54.988853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:25:55.090702 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:25:55.090670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:25:55.423339 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:25:55.423307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:26:21.376393 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:26:21.376356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:26:40.632452 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:26:40.632415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:26:40.731652 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:26:40.731610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:26:48.332882 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:26:48.332851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:26:49.457739 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:26:49.457683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:27:22.527268 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:22.527231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:27:30.321824 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:30.321737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:27:31.198428 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:31.198394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:27:32.177062 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.177022 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7607-140230637451824"] Jul 09 20:27:32.177503 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.177489 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:27:32.177555 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.177506 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:27:32.177653 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.177642 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="364786f7-af28-4049-9d81-9cf67fc2b347" containerName="istio-proxy" Jul 09 20:27:32.180721 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.180705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:32.184261 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.184233 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-qvtcl\"" Jul 09 20:27:32.187547 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.187523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230637451824"] Jul 09 20:27:32.217712 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.217679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwr7r\" (UniqueName: \"kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r\") pod \"test-curl-7607-140230637451824\" (UID: \"116d988c-6faa-4869-a322-d89d0f0f5ceb\") " pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:32.318859 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.318825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwr7r\" (UniqueName: \"kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r\") pod \"test-curl-7607-140230637451824\" (UID: \"116d988c-6faa-4869-a322-d89d0f0f5ceb\") " pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:32.327994 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.327973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwr7r\" (UniqueName: \"kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r\") pod \"test-curl-7607-140230637451824\" (UID: \"116d988c-6faa-4869-a322-d89d0f0f5ceb\") " pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:32.491203 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.491113 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:32.631650 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.631621 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230637451824"] Jul 09 20:27:32.632539 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:27:32.632514 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod116d988c_6faa_4869_a322_d89d0f0f5ceb.slice/crio-4d334fcc7ac31d3f1aa28c6348eb2d3f581051e47a79a06be342d4ecec13bd1d WatchSource:0}: Error finding container 4d334fcc7ac31d3f1aa28c6348eb2d3f581051e47a79a06be342d4ecec13bd1d: Status 404 returned error can't find the container with id 4d334fcc7ac31d3f1aa28c6348eb2d3f581051e47a79a06be342d4ecec13bd1d Jul 09 20:27:32.633822 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:32.633798 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 20:27:33.202777 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:33.202731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230637451824" event={"ID":"116d988c-6faa-4869-a322-d89d0f0f5ceb","Type":"ContainerStarted","Data":"4d334fcc7ac31d3f1aa28c6348eb2d3f581051e47a79a06be342d4ecec13bd1d"} Jul 09 20:27:34.209347 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:34.209314 2562 generic.go:356] "Generic (PLEG): container finished" podID="116d988c-6faa-4869-a322-d89d0f0f5ceb" containerID="8a78039f15ecfeb527fb771c628cbfe02e881d0a62acb094320e8fc87dceb226" exitCode=0 Jul 09 20:27:34.209778 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:34.209402 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230637451824" event={"ID":"116d988c-6faa-4869-a322-d89d0f0f5ceb","Type":"ContainerDied","Data":"8a78039f15ecfeb527fb771c628cbfe02e881d0a62acb094320e8fc87dceb226"} Jul 09 20:27:35.363472 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.363445 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:35.384701 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.384676 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7607-140230637451824_116d988c-6faa-4869-a322-d89d0f0f5ceb/test-curl-7607-140230637451824/0.log" Jul 09 20:27:35.410206 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.410163 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7607-140230637451824"] Jul 09 20:27:35.413868 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.413843 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7607-140230637451824"] Jul 09 20:27:35.447419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.447390 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwr7r\" (UniqueName: \"kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r\") pod \"116d988c-6faa-4869-a322-d89d0f0f5ceb\" (UID: \"116d988c-6faa-4869-a322-d89d0f0f5ceb\") " Jul 09 20:27:35.449695 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.449662 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r" (OuterVolumeSpecName: "kube-api-access-hwr7r") pod "116d988c-6faa-4869-a322-d89d0f0f5ceb" (UID: "116d988c-6faa-4869-a322-d89d0f0f5ceb"). InnerVolumeSpecName "kube-api-access-hwr7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:27:35.543794 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.543758 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7607-140230640411440"] Jul 09 20:27:35.544172 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.544159 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116d988c-6faa-4869-a322-d89d0f0f5ceb" containerName="test-curl-7607-140230637451824" Jul 09 20:27:35.544234 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.544174 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="116d988c-6faa-4869-a322-d89d0f0f5ceb" containerName="test-curl-7607-140230637451824" Jul 09 20:27:35.544270 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.544246 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="116d988c-6faa-4869-a322-d89d0f0f5ceb" containerName="test-curl-7607-140230637451824" Jul 09 20:27:35.547506 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.547489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:35.548015 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.547996 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwr7r\" (UniqueName: \"kubernetes.io/projected/116d988c-6faa-4869-a322-d89d0f0f5ceb-kube-api-access-hwr7r\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:27:35.555864 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.555839 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230640411440"] Jul 09 20:27:35.648566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.648538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsk4g\" (UniqueName: \"kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g\") pod \"test-curl-7607-140230640411440\" (UID: \"d8b2c45d-2789-411e-9862-5ed336915175\") " pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:35.749918 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.749870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsk4g\" (UniqueName: \"kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g\") pod \"test-curl-7607-140230640411440\" (UID: \"d8b2c45d-2789-411e-9862-5ed336915175\") " pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:35.759366 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.759336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsk4g\" (UniqueName: \"kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g\") pod \"test-curl-7607-140230640411440\" (UID: \"d8b2c45d-2789-411e-9862-5ed336915175\") " pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:35.857648 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.857617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:35.957201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.957172 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116d988c-6faa-4869-a322-d89d0f0f5ceb" path="/var/lib/kubelet/pods/116d988c-6faa-4869-a322-d89d0f0f5ceb/volumes" Jul 09 20:27:35.998643 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:35.998614 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230640411440"] Jul 09 20:27:36.000533 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:27:36.000510 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8b2c45d_2789_411e_9862_5ed336915175.slice/crio-08f9b5b528ecf9bd62f366c201fd20d355c96b8f428c340a771138c13dd44cd9 WatchSource:0}: Error finding container 08f9b5b528ecf9bd62f366c201fd20d355c96b8f428c340a771138c13dd44cd9: Status 404 returned error can't find the container with id 08f9b5b528ecf9bd62f366c201fd20d355c96b8f428c340a771138c13dd44cd9 Jul 09 20:27:36.222875 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:36.222789 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230640411440" event={"ID":"d8b2c45d-2789-411e-9862-5ed336915175","Type":"ContainerStarted","Data":"08f9b5b528ecf9bd62f366c201fd20d355c96b8f428c340a771138c13dd44cd9"} Jul 09 20:27:36.224065 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:36.224044 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230637451824" Jul 09 20:27:36.224195 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:36.224073 2562 scope.go:117] "RemoveContainer" containerID="8a78039f15ecfeb527fb771c628cbfe02e881d0a62acb094320e8fc87dceb226" Jul 09 20:27:37.231668 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:37.231631 2562 generic.go:356] "Generic (PLEG): container finished" podID="d8b2c45d-2789-411e-9862-5ed336915175" containerID="ccefeb9dbb5c9d97ebdc0ca9319de33e1c758bcd562c634550e88849ec82fcf7" exitCode=0 Jul 09 20:27:37.232127 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:37.231722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230640411440" event={"ID":"d8b2c45d-2789-411e-9862-5ed336915175","Type":"ContainerDied","Data":"ccefeb9dbb5c9d97ebdc0ca9319de33e1c758bcd562c634550e88849ec82fcf7"} Jul 09 20:27:38.382196 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.382170 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:38.402502 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.402474 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7607-140230640411440_d8b2c45d-2789-411e-9862-5ed336915175/test-curl-7607-140230640411440/0.log" Jul 09 20:27:38.430830 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.430805 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7607-140230640411440"] Jul 09 20:27:38.433231 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.433211 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7607-140230640411440"] Jul 09 20:27:38.475255 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.475222 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsk4g\" (UniqueName: \"kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g\") pod \"d8b2c45d-2789-411e-9862-5ed336915175\" (UID: \"d8b2c45d-2789-411e-9862-5ed336915175\") " Jul 09 20:27:38.477414 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.477385 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g" (OuterVolumeSpecName: "kube-api-access-fsk4g") pod "d8b2c45d-2789-411e-9862-5ed336915175" (UID: "d8b2c45d-2789-411e-9862-5ed336915175"). InnerVolumeSpecName "kube-api-access-fsk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:27:38.576250 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.576171 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsk4g\" (UniqueName: \"kubernetes.io/projected/d8b2c45d-2789-411e-9862-5ed336915175-kube-api-access-fsk4g\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:27:38.615265 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.615237 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:38.615722 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.615709 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b2c45d-2789-411e-9862-5ed336915175" containerName="test-curl-7607-140230640411440" Jul 09 20:27:38.615780 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.615723 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b2c45d-2789-411e-9862-5ed336915175" containerName="test-curl-7607-140230640411440" Jul 09 20:27:38.615817 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.615783 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b2c45d-2789-411e-9862-5ed336915175" containerName="test-curl-7607-140230640411440" Jul 09 20:27:38.618740 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.618725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:38.625006 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.624979 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:38.677056 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.677030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8sk\" (UniqueName: \"kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk\") pod \"test-curl-7607-140230641700416\" (UID: \"c34d62c3-b543-4740-8632-c750b1293c88\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:38.777697 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.777658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8sk\" (UniqueName: \"kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk\") pod \"test-curl-7607-140230641700416\" (UID: \"c34d62c3-b543-4740-8632-c750b1293c88\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:38.787031 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.787007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8sk\" (UniqueName: \"kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk\") pod \"test-curl-7607-140230641700416\" (UID: \"c34d62c3-b543-4740-8632-c750b1293c88\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:38.928900 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:38.928867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:39.068724 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:39.068701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:39.070082 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:27:39.070058 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc34d62c3_b543_4740_8632_c750b1293c88.slice/crio-8990a4c8507c2c48e2da36f3f9bdd870aac572534528138dc3fb8e9723b8a39c WatchSource:0}: Error finding container 8990a4c8507c2c48e2da36f3f9bdd870aac572534528138dc3fb8e9723b8a39c: Status 404 returned error can't find the container with id 8990a4c8507c2c48e2da36f3f9bdd870aac572534528138dc3fb8e9723b8a39c Jul 09 20:27:39.243669 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:39.243565 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:39.243823 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:39.243575 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f9b5b528ecf9bd62f366c201fd20d355c96b8f428c340a771138c13dd44cd9" Jul 09 20:27:39.244881 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:39.244854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700416" event={"ID":"c34d62c3-b543-4740-8632-c750b1293c88","Type":"ContainerStarted","Data":"8990a4c8507c2c48e2da36f3f9bdd870aac572534528138dc3fb8e9723b8a39c"} Jul 09 20:27:39.254093 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:27:39.254066 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7607-140230640411440\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="d8b2c45d-2789-411e-9862-5ed336915175" pod="openshift-ingress/test-curl-7607-140230640411440" Jul 09 20:27:39.953370 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:39.953339 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b2c45d-2789-411e-9862-5ed336915175" path="/var/lib/kubelet/pods/d8b2c45d-2789-411e-9862-5ed336915175/volumes" Jul 09 20:27:40.251861 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:40.251775 2562 generic.go:356] "Generic (PLEG): container finished" podID="c34d62c3-b543-4740-8632-c750b1293c88" containerID="81c439cb220de4f48c1cd0aa928b5780e1c630958dd31b3767bc209183e35330" exitCode=0 Jul 09 20:27:40.251997 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:40.251862 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700416" event={"ID":"c34d62c3-b543-4740-8632-c750b1293c88","Type":"ContainerDied","Data":"81c439cb220de4f48c1cd0aa928b5780e1c630958dd31b3767bc209183e35330"} Jul 09 20:27:41.408693 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.408664 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.430764 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.430731 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7607-140230641700416_c34d62c3-b543-4740-8632-c750b1293c88/test-curl-7607-140230641700416/0.log" Jul 09 20:27:41.456971 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.456930 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:41.461717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.461680 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:41.500827 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.500792 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw8sk\" (UniqueName: \"kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk\") pod \"c34d62c3-b543-4740-8632-c750b1293c88\" (UID: \"c34d62c3-b543-4740-8632-c750b1293c88\") " Jul 09 20:27:41.503088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.503061 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk" (OuterVolumeSpecName: "kube-api-access-vw8sk") pod "c34d62c3-b543-4740-8632-c750b1293c88" (UID: "c34d62c3-b543-4740-8632-c750b1293c88"). InnerVolumeSpecName "kube-api-access-vw8sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:27:41.600136 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.600097 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:41.600616 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.600598 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34d62c3-b543-4740-8632-c750b1293c88" containerName="test-curl-7607-140230641700416" Jul 09 20:27:41.600727 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.600618 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34d62c3-b543-4740-8632-c750b1293c88" containerName="test-curl-7607-140230641700416" Jul 09 20:27:41.600727 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.600709 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34d62c3-b543-4740-8632-c750b1293c88" containerName="test-curl-7607-140230641700416" Jul 09 20:27:41.602525 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.602493 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8sk\" (UniqueName: \"kubernetes.io/projected/c34d62c3-b543-4740-8632-c750b1293c88-kube-api-access-vw8sk\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:27:41.604553 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.604518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.613703 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.613679 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:41.703444 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.703404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5zt\" (UniqueName: \"kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt\") pod \"test-curl-7607-140230641700416\" (UID: \"91db3d75-6c1f-4c81-a7ee-508264635db1\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.804837 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.804748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5zt\" (UniqueName: \"kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt\") pod \"test-curl-7607-140230641700416\" (UID: \"91db3d75-6c1f-4c81-a7ee-508264635db1\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.813466 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.813432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5zt\" (UniqueName: \"kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt\") pod \"test-curl-7607-140230641700416\" (UID: \"91db3d75-6c1f-4c81-a7ee-508264635db1\") " pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.919388 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.919353 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:41.955155 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:41.955111 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34d62c3-b543-4740-8632-c750b1293c88" path="/var/lib/kubelet/pods/c34d62c3-b543-4740-8632-c750b1293c88/volumes" Jul 09 20:27:42.063498 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:42.063469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:42.064727 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:27:42.064697 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91db3d75_6c1f_4c81_a7ee_508264635db1.slice/crio-9e82473f3cdb9f2f028845075439f3f9b0a53d614c5941eca51657067fafdb42 WatchSource:0}: Error finding container 9e82473f3cdb9f2f028845075439f3f9b0a53d614c5941eca51657067fafdb42: Status 404 returned error can't find the container with id 9e82473f3cdb9f2f028845075439f3f9b0a53d614c5941eca51657067fafdb42 Jul 09 20:27:42.265180 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:42.265130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700416" event={"ID":"91db3d75-6c1f-4c81-a7ee-508264635db1","Type":"ContainerStarted","Data":"9e82473f3cdb9f2f028845075439f3f9b0a53d614c5941eca51657067fafdb42"} Jul 09 20:27:42.266423 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:42.266401 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:42.266572 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:42.266428 2562 scope.go:117] "RemoveContainer" containerID="81c439cb220de4f48c1cd0aa928b5780e1c630958dd31b3767bc209183e35330" Jul 09 20:27:43.273753 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:43.273722 2562 generic.go:356] "Generic (PLEG): container finished" podID="91db3d75-6c1f-4c81-a7ee-508264635db1" containerID="cacef3e8c01ae2e0b2a91f1bebb6b6dcdc50557c93d06b4ed030291ea277e745" exitCode=0 Jul 09 20:27:43.274235 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:43.273813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700416" event={"ID":"91db3d75-6c1f-4c81-a7ee-508264635db1","Type":"ContainerDied","Data":"cacef3e8c01ae2e0b2a91f1bebb6b6dcdc50557c93d06b4ed030291ea277e745"} Jul 09 20:27:44.413872 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.413842 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:44.435974 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.435948 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7607-140230641700416_91db3d75-6c1f-4c81-a7ee-508264635db1/test-curl-7607-140230641700416/0.log" Jul 09 20:27:44.464005 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.463972 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:44.467845 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.467820 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700416"] Jul 09 20:27:44.530693 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.530660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv5zt\" (UniqueName: \"kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt\") pod \"91db3d75-6c1f-4c81-a7ee-508264635db1\" (UID: \"91db3d75-6c1f-4c81-a7ee-508264635db1\") " Jul 09 20:27:44.532968 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.532909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt" (OuterVolumeSpecName: "kube-api-access-mv5zt") pod "91db3d75-6c1f-4c81-a7ee-508264635db1" (UID: "91db3d75-6c1f-4c81-a7ee-508264635db1"). InnerVolumeSpecName "kube-api-access-mv5zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:27:44.595066 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.595033 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7607-140230641700272"] Jul 09 20:27:44.595469 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.595456 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91db3d75-6c1f-4c81-a7ee-508264635db1" containerName="test-curl-7607-140230641700416" Jul 09 20:27:44.595513 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.595472 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="91db3d75-6c1f-4c81-a7ee-508264635db1" containerName="test-curl-7607-140230641700416" Jul 09 20:27:44.595548 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.595540 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="91db3d75-6c1f-4c81-a7ee-508264635db1" containerName="test-curl-7607-140230641700416" Jul 09 20:27:44.598293 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.598278 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:44.605081 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.605057 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700272"] Jul 09 20:27:44.631514 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.631489 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5zt\" (UniqueName: \"kubernetes.io/projected/91db3d75-6c1f-4c81-a7ee-508264635db1-kube-api-access-mv5zt\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:27:44.732889 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.732853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsc2\" (UniqueName: \"kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2\") pod \"test-curl-7607-140230641700272\" (UID: \"9e36a123-a1d1-458d-8ff2-f8f451210c4a\") " pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:44.833996 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.833912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsc2\" (UniqueName: \"kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2\") pod \"test-curl-7607-140230641700272\" (UID: \"9e36a123-a1d1-458d-8ff2-f8f451210c4a\") " pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:44.843299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.843272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsc2\" (UniqueName: \"kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2\") pod \"test-curl-7607-140230641700272\" (UID: \"9e36a123-a1d1-458d-8ff2-f8f451210c4a\") " pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:44.908202 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:44.908174 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:45.045979 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:45.045954 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700272"] Jul 09 20:27:45.047548 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:27:45.047514 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e36a123_a1d1_458d_8ff2_f8f451210c4a.slice/crio-1b688943cc4c779414cfb5238035e93e251206f062e0a12e3aa32a9dc8c89024 WatchSource:0}: Error finding container 1b688943cc4c779414cfb5238035e93e251206f062e0a12e3aa32a9dc8c89024: Status 404 returned error can't find the container with id 1b688943cc4c779414cfb5238035e93e251206f062e0a12e3aa32a9dc8c89024 Jul 09 20:27:45.287718 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:45.287676 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:45.287926 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:45.287728 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e82473f3cdb9f2f028845075439f3f9b0a53d614c5941eca51657067fafdb42" Jul 09 20:27:45.288939 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:45.288912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700272" event={"ID":"9e36a123-a1d1-458d-8ff2-f8f451210c4a","Type":"ContainerStarted","Data":"1b688943cc4c779414cfb5238035e93e251206f062e0a12e3aa32a9dc8c89024"} Jul 09 20:27:45.298387 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:27:45.298363 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7607-140230641700416\" is forbidden: User \"system:node:ip-10-0-139-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-128.ec2.internal' and this object" podUID="91db3d75-6c1f-4c81-a7ee-508264635db1" pod="openshift-ingress/test-curl-7607-140230641700416" Jul 09 20:27:45.953683 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:45.953651 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91db3d75-6c1f-4c81-a7ee-508264635db1" path="/var/lib/kubelet/pods/91db3d75-6c1f-4c81-a7ee-508264635db1/volumes" Jul 09 20:27:46.296380 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:46.296289 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e36a123-a1d1-458d-8ff2-f8f451210c4a" containerID="2174bae478202f1e3eb510f6b34fa3d20f925048ee3d4f1023768e58dbec6682" exitCode=0 Jul 09 20:27:46.296544 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:46.296382 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7607-140230641700272" event={"ID":"9e36a123-a1d1-458d-8ff2-f8f451210c4a","Type":"ContainerDied","Data":"2174bae478202f1e3eb510f6b34fa3d20f925048ee3d4f1023768e58dbec6682"} Jul 09 20:27:47.450889 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.450864 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:47.472320 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.472292 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7607-140230641700272_9e36a123-a1d1-458d-8ff2-f8f451210c4a/test-curl-7607-140230641700272/0.log" Jul 09 20:27:47.499293 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.499259 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700272"] Jul 09 20:27:47.504991 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.504967 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7607-140230641700272"] Jul 09 20:27:47.555638 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.555557 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmsc2\" (UniqueName: \"kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2\") pod \"9e36a123-a1d1-458d-8ff2-f8f451210c4a\" (UID: \"9e36a123-a1d1-458d-8ff2-f8f451210c4a\") " Jul 09 20:27:47.557826 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.557805 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2" (OuterVolumeSpecName: "kube-api-access-wmsc2") pod "9e36a123-a1d1-458d-8ff2-f8f451210c4a" (UID: "9e36a123-a1d1-458d-8ff2-f8f451210c4a"). InnerVolumeSpecName "kube-api-access-wmsc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:27:47.657299 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.657260 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsc2\" (UniqueName: \"kubernetes.io/projected/9e36a123-a1d1-458d-8ff2-f8f451210c4a-kube-api-access-wmsc2\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:27:47.954105 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:47.954069 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e36a123-a1d1-458d-8ff2-f8f451210c4a" path="/var/lib/kubelet/pods/9e36a123-a1d1-458d-8ff2-f8f451210c4a/volumes" Jul 09 20:27:48.309261 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:48.309178 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7607-140230641700272" Jul 09 20:27:48.309261 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:48.309219 2562 scope.go:117] "RemoveContainer" containerID="2174bae478202f1e3eb510f6b34fa3d20f925048ee3d4f1023768e58dbec6682" Jul 09 20:27:51.551980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:51.551944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:27:51.652491 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:27:51.652462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:01.594554 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:01.594501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:06.159510 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:06.159478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:12.060076 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:12.060043 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:12.127227 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:12.127194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:16.833637 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:16.833602 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:28:16.834081 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:16.833829 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" containerID="cri-o://a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7" gracePeriod=30 Jul 09 20:28:16.859689 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:16.859658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:17.098333 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:17.098248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:17.855029 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:17.854999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:17.953943 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:17.953904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:22.098453 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.098395 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:28:22.174122 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174087 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174122 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174129 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfknv\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174179 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174195 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174212 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174250 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174317 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174374 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo\") pod \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\" (UID: \"a36570b9-7c85-4ea5-8ba8-2c961da56aec\") " Jul 09 20:28:22.174763 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174451 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:22.174763 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174568 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:22.174763 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174627 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.174918 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174881 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:28:22.175036 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.174960 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:22.175228 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.175098 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data" (OuterVolumeSpecName: "istio-data") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:22.176821 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.176799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:28:22.176925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.176828 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:22.177151 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.177122 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv" (OuterVolumeSpecName: "kube-api-access-cfknv") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "kube-api-access-cfknv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:28:22.177201 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.177165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token" (OuterVolumeSpecName: "istio-token") pod "a36570b9-7c85-4ea5-8ba8-2c961da56aec" (UID: "a36570b9-7c85-4ea5-8ba8-2c961da56aec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:28:22.275054 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275020 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275054 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275048 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275054 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275060 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275071 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfknv\" (UniqueName: \"kubernetes.io/projected/a36570b9-7c85-4ea5-8ba8-2c961da56aec-kube-api-access-cfknv\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275085 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275096 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275106 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36570b9-7c85-4ea5-8ba8-2c961da56aec-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.275295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.275117 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36570b9-7c85-4ea5-8ba8-2c961da56aec-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:22.508107 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.508067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:22.531103 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.531072 2562 generic.go:356] "Generic (PLEG): container finished" podID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerID="a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7" exitCode=0 Jul 09 20:28:22.531314 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.531122 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" Jul 09 20:28:22.531314 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.531161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerDied","Data":"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7"} Jul 09 20:28:22.531314 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.531226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl" event={"ID":"a36570b9-7c85-4ea5-8ba8-2c961da56aec","Type":"ContainerDied","Data":"73cf9aec29270e585bebf4502900ab0011d0336a4da63c52d014edd047466db0"} Jul 09 20:28:22.531314 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.531256 2562 scope.go:117] "RemoveContainer" containerID="a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7" Jul 09 20:28:22.541519 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.541503 2562 scope.go:117] "RemoveContainer" containerID="80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5" Jul 09 20:28:22.550097 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.550082 2562 scope.go:117] "RemoveContainer" containerID="a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7" Jul 09 20:28:22.550321 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:28:22.550302 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7\": container with ID starting with a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7 not found: ID does not exist" containerID="a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7" Jul 09 20:28:22.550385 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.550328 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7"} err="failed to get container status \"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7\": rpc error: code = NotFound desc = could not find container \"a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7\": container with ID starting with a5a0e3a5b4cc35d0557bb24726ff76eee0fcf2a3dccfdf9bdc2099616b3d3fe7 not found: ID does not exist" Jul 09 20:28:22.550385 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.550345 2562 scope.go:117] "RemoveContainer" containerID="80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5" Jul 09 20:28:22.550573 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:28:22.550556 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5\": container with ID starting with 80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5 not found: ID does not exist" containerID="80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5" Jul 09 20:28:22.550720 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.550599 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5"} err="failed to get container status \"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5\": rpc error: code = NotFound desc = could not find container \"80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5\": container with ID starting with 80ab62dac42ab852c0b6753c1b4c2461147765afbc38bae1bfd22dddaf5895b5 not found: ID does not exist" Jul 09 20:28:22.555611 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.555572 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:28:22.563223 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.563199 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7b406b-openshift-default-bc55558c5-l8rjl"] Jul 09 20:28:22.604533 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:22.604490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:23.954116 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:23.954076 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" path="/var/lib/kubelet/pods/a36570b9-7c85-4ea5-8ba8-2c961da56aec/volumes" Jul 09 20:28:27.756066 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:27.756031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:27.856700 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:27.856673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:28.015344 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:28.015255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:32.319020 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:32.318986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:38.236167 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:38.236135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:38.337365 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:38.337330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:43.012331 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:43.012297 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:28:43.012796 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:43.012531 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" containerID="cri-o://815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac" gracePeriod=30 Jul 09 20:28:43.038987 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:43.038961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:43.279841 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:43.279737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:28:43.718282 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:43.718251 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84445d6688-df7rg_7ac93773-1f97-4770-a62b-a0d7f563fa1c/authorino/0.log" Jul 09 20:28:46.472918 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:46.472878 2562 patch_prober.go:28] interesting pod/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 20:28:46.473319 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:46.472946 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 20:28:48.273990 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.273927 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:28:48.415061 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415097 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415123 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415229 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415174 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415227 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2kw4\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415257 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415306 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415331 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415367 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415359 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert\") pod \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\" (UID: \"5553a71a-cd5a-4d13-a4f1-486b264c8f31\") " Jul 09 20:28:48.415652 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415456 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:48.415793 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415749 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-credential-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.415886 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415854 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:48.416107 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.415792 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 20:28:48.416333 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.416145 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:48.416785 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.416756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data" (OuterVolumeSpecName: "istio-data") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:48.417547 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.417515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token" (OuterVolumeSpecName: "istio-token") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:28:48.417547 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.417520 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 20:28:48.417745 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.417605 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 20:28:48.417745 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.417729 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4" (OuterVolumeSpecName: "kube-api-access-g2kw4") pod "5553a71a-cd5a-4d13-a4f1-486b264c8f31" (UID: "5553a71a-cd5a-4d13-a4f1-486b264c8f31"). InnerVolumeSpecName "kube-api-access-g2kw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 20:28:48.516566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516534 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kw4\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-kube-api-access-g2kw4\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516562 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-socket\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516574 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-data\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516603 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-workload-certs\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516611 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istiod-ca-cert\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516619 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-token\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516626 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-podinfo\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.516809 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.516633 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5553a71a-cd5a-4d13-a4f1-486b264c8f31-istio-envoy\") on node \"ip-10-0-139-128.ec2.internal\" DevicePath \"\"" Jul 09 20:28:48.686113 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.686076 2562 generic.go:356] "Generic (PLEG): container finished" podID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerID="815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac" exitCode=0 Jul 09 20:28:48.686320 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.686130 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" Jul 09 20:28:48.686320 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.686165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerDied","Data":"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac"} Jul 09 20:28:48.686320 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.686237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp" event={"ID":"5553a71a-cd5a-4d13-a4f1-486b264c8f31","Type":"ContainerDied","Data":"6fdf56ff8c4981ca4246fef8b0f1854939871774d4fed02728b5ec44d1a521b4"} Jul 09 20:28:48.686320 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.686269 2562 scope.go:117] "RemoveContainer" containerID="815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac" Jul 09 20:28:48.696137 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.696121 2562 scope.go:117] "RemoveContainer" containerID="68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df" Jul 09 20:28:48.704434 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.704417 2562 scope.go:117] "RemoveContainer" containerID="815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac" Jul 09 20:28:48.704719 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:28:48.704701 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac\": container with ID starting with 815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac not found: ID does not exist" containerID="815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac" Jul 09 20:28:48.704789 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.704728 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac"} err="failed to get container status \"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac\": rpc error: code = NotFound desc = could not find container \"815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac\": container with ID starting with 815e6321ac44ace0729c9a7f1e0082723eb0de0316218322493dbe6d8426d9ac not found: ID does not exist" Jul 09 20:28:48.704789 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.704745 2562 scope.go:117] "RemoveContainer" containerID="68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df" Jul 09 20:28:48.704962 ip-10-0-139-128 kubenswrapper[2562]: E0709 20:28:48.704942 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df\": container with ID starting with 68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df not found: ID does not exist" containerID="68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df" Jul 09 20:28:48.705000 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.704970 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df"} err="failed to get container status \"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df\": rpc error: code = NotFound desc = could not find container \"68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df\": container with ID starting with 68ce9a86840c5ec0dd1e8cfe2df3e2cf9ea643f440b812f2e48eea5e6e5d19df not found: ID does not exist" Jul 09 20:28:48.709253 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.709231 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:28:48.713673 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.713655 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-3452a7-openshift-default-646bdb955-5wwtp"] Jul 09 20:28:48.836652 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.836628 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4pn8q_e3db049e-97d4-4c00-a24f-ba5272bfc88f/manager/0.log" Jul 09 20:28:48.963091 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:48.962984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-khxfn_6f65dcb6-a83e-4aed-a5f0-abd60438e4f2/manager/0.log" Jul 09 20:28:49.089991 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.089960 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6885c5f9d9-sx24t_d1ab6e2f-286f-40cc-860e-894515cd7a86/manager/0.log" Jul 09 20:28:49.210925 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.210896 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g5fl4_5faa2b79-08e6-46e1-a4d8-94ab0068a98b/server/0.log" Jul 09 20:28:49.351997 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.351927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ncn95_7cedd294-177f-46b8-996c-5150e85d805b/manager/0.log" Jul 09 20:28:49.619208 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.619128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-776898495d-vxjqn_174887e9-7aaa-4a90-9538-fc9a29d000d4/copy-manifests/0.log" Jul 09 20:28:49.627566 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.627539 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-776898495d-vxjqn_174887e9-7aaa-4a90-9538-fc9a29d000d4/manager/0.log" Jul 09 20:28:49.955382 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:49.955347 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" path="/var/lib/kubelet/pods/5553a71a-cd5a-4d13-a4f1-486b264c8f31/volumes" Jul 09 20:28:50.162549 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:50.162521 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f7dd46c4-h6r9g_9cedb07c-f373-4c45-93b2-13a10fedbc2e/maas-api/0.log" Jul 09 20:28:50.286115 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:50.286040 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727120-b28rw_50ae6e11-9cae-41e5-9d2a-636065771db1/cleanup/0.log" Jul 09 20:28:50.527480 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:50.527451 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-581685c1-p5nhx_0bba33cd-e80b-46c3-82e4-52665f9ff25a/revoke-keys/0.log" Jul 09 20:28:51.366789 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:51.366759 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-71d42f8b-zgjb2_957bfd2b-b195-4b95-88bf-bcfca0b02088/revoke-keys/0.log" Jul 09 20:28:51.857008 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:51.856980 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sg56x_680979e4-101f-4adb-a529-84424a175b22/postgres/0.log" Jul 09 20:28:52.699022 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.698978 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/extract/0.log" Jul 09 20:28:52.705541 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.705513 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/util/0.log" Jul 09 20:28:52.712565 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.712543 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/pull/0.log" Jul 09 20:28:52.831128 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.831099 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/util/0.log" Jul 09 20:28:52.838224 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.838204 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/pull/0.log" Jul 09 20:28:52.844831 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.844811 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/extract/0.log" Jul 09 20:28:52.959048 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.958969 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/util/0.log" Jul 09 20:28:52.966295 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.966270 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/pull/0.log" Jul 09 20:28:52.973301 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:52.973282 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/extract/0.log" Jul 09 20:28:53.093334 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.093306 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/pull/0.log" Jul 09 20:28:53.101707 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.101678 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/extract/0.log" Jul 09 20:28:53.109153 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.109131 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/util/0.log" Jul 09 20:28:53.239733 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.239650 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84445d6688-df7rg_7ac93773-1f97-4770-a62b-a0d7f563fa1c/authorino/0.log" Jul 09 20:28:53.364895 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.364865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wqjh7_4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d/manager/0.log" Jul 09 20:28:53.491166 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.491078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-g6n9w_b56839c4-2f35-40bf-a73f-5396bea70761/manager/0.log" Jul 09 20:28:53.611942 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.611918 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8l4hp_a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42/kuadrant-console-plugin/0.log" Jul 09 20:28:53.736139 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:53.736111 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5p2h8_c7c9a28e-de41-43e3-bce5-2dcbb62b32c9/registry-server/0.log" Jul 09 20:28:54.146725 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:54.146693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zqmlt_783925ea-937f-4803-9c6a-c4796a98a073/manager/0.log" Jul 09 20:28:54.525158 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:54.525078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vnbbh_e0a9fec9-fa38-4c3b-91d5-7d03a6026c97/istio-proxy/0.log" Jul 09 20:28:54.902121 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:54.902096 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5f4bc74d-x2hz8_24f55663-8dfb-43b4-a2e4-06fb45a5a38b/kube-auth-proxy/0.log" Jul 09 20:28:55.025975 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:55.025951 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/1.log" Jul 09 20:28:55.150599 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:55.150558 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lch5w_3d6a813a-2aae-4908-9561-5f9b79fc8857/payload-pre-processing/0.log" Jul 09 20:28:55.270753 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:55.270674 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7ptzq_2ee1edf3-ea29-47fb-be62-cc8260de47a1/payload-processing/0.log" Jul 09 20:28:56.129244 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:28:56.129214 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblcgpg_71f2197e-0f37-410d-875c-2c46b4305f7b/main/0.log" Jul 09 20:29:00.168147 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:00.168120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:29:00.269695 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:00.269666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:29:02.969325 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:02.969296 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9kgj4_8afa780b-2966-4674-aa12-fe1b7b6e4a56/global-pull-secret-syncer/0.log" Jul 09 20:29:03.060561 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:03.060535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" Jul 09 20:29:03.207858 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:03.207828 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-128.ec2.internal_934de4e9f3c9b88ff4abff78cc84b221/haproxy/0.log" Jul 09 20:29:08.169512 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.169479 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/extract/0.log" Jul 09 20:29:08.197717 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.197684 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/util/0.log" Jul 09 20:29:08.226983 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.226959 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mmzj6_7697830a-cdf4-42bb-839c-699a74718ce5/pull/0.log" Jul 09 20:29:08.254250 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.254226 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/extract/0.log" Jul 09 20:29:08.281645 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.281621 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/util/0.log" Jul 09 20:29:08.309860 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.309840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hxq5m_98a4ed1a-9efd-4976-8732-799ab79c5e6b/pull/0.log" Jul 09 20:29:08.337458 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.337437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/extract/0.log" Jul 09 20:29:08.361137 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.361113 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/util/0.log" Jul 09 20:29:08.387602 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.387558 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kmjlm_0a5a4166-2d55-414f-a876-28017499e692/pull/0.log" Jul 09 20:29:08.417919 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.417900 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/extract/0.log" Jul 09 20:29:08.442980 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.442917 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/util/0.log" Jul 09 20:29:08.465236 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.465212 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12l75b_f349fa74-6d5b-4d91-aeec-6acf384487c6/pull/0.log" Jul 09 20:29:08.781812 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.781718 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84445d6688-df7rg_7ac93773-1f97-4770-a62b-a0d7f563fa1c/authorino/0.log" Jul 09 20:29:08.809394 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.809361 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wqjh7_4758cc9a-897f-4f09-b6d1-c32b3ba0bc5d/manager/0.log" Jul 09 20:29:08.838815 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.838787 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-g6n9w_b56839c4-2f35-40bf-a73f-5396bea70761/manager/0.log" Jul 09 20:29:08.864788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.864760 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8l4hp_a403c0b2-4f9b-42b8-bcd3-cc5049f9ac42/kuadrant-console-plugin/0.log" Jul 09 20:29:08.905536 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:08.905504 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5p2h8_c7c9a28e-de41-43e3-bce5-2dcbb62b32c9/registry-server/0.log" Jul 09 20:29:09.169337 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:09.169305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zqmlt_783925ea-937f-4803-9c6a-c4796a98a073/manager/0.log" Jul 09 20:29:09.829495 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:09.829464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56tk9_2c95275d-ea82-412c-bdc1-101e3b1f7f73/konnectivity-agent/0.log" limit=1024 Jul 09 20:29:11.202254 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:11.202221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nwl5_6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf/node-exporter/0.log" Jul 09 20:29:11.222873 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:11.222845 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nwl5_6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf/kube-rbac-proxy/0.log" Jul 09 20:29:11.245000 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:11.244977 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nwl5_6d22e7b9-cf4a-4afa-8ca3-50bda477b3cf/init-textfile/0.log" Jul 09 20:29:13.192001 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.191967 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv"] Jul 09 20:29:13.192378 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192347 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" Jul 09 20:29:13.192378 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192366 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" Jul 09 20:29:13.192378 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192377 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36a123-a1d1-458d-8ff2-f8f451210c4a" containerName="test-curl-7607-140230641700272" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192384 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36a123-a1d1-458d-8ff2-f8f451210c4a" containerName="test-curl-7607-140230641700272" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192399 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192424 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192429 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192437 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" Jul 09 20:29:13.192488 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192442 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" Jul 09 20:29:13.192839 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192497 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36570b9-7c85-4ea5-8ba8-2c961da56aec" containerName="istio-proxy" Jul 09 20:29:13.192839 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192507 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5553a71a-cd5a-4d13-a4f1-486b264c8f31" containerName="istio-proxy" Jul 09 20:29:13.192839 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.192515 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36a123-a1d1-458d-8ff2-f8f451210c4a" containerName="test-curl-7607-140230641700272" Jul 09 20:29:13.195679 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.195659 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.198531 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.198509 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l62zn\"/\"default-dockercfg-tnls4\"" Jul 09 20:29:13.198654 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.198509 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l62zn\"/\"openshift-service-ca.crt\"" Jul 09 20:29:13.199630 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.199614 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l62zn\"/\"kube-root-ca.crt\"" Jul 09 20:29:13.203942 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.203906 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv"] Jul 09 20:29:13.226676 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.226647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-proc\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.226788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.226698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-sys\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.226788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.226714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-lib-modules\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.226788 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.226779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-podres\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.226894 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.226854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvs89\" (UniqueName: \"kubernetes.io/projected/df8a0c21-1091-426b-b49b-eb0d52af8db2-kube-api-access-bvs89\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328277 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-podres\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvs89\" (UniqueName: \"kubernetes.io/projected/df8a0c21-1091-426b-b49b-eb0d52af8db2-kube-api-access-bvs89\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328335 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-proc\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-sys\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-lib-modules\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328435 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328417 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-podres\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328630 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-proc\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328630 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-sys\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.328630 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.328519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df8a0c21-1091-426b-b49b-eb0d52af8db2-lib-modules\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.337780 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.337759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvs89\" (UniqueName: \"kubernetes.io/projected/df8a0c21-1091-426b-b49b-eb0d52af8db2-kube-api-access-bvs89\") pod \"perf-node-gather-daemonset-zgptv\" (UID: \"df8a0c21-1091-426b-b49b-eb0d52af8db2\") " pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.506255 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.506164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.639962 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.639934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv"] Jul 09 20:29:13.641195 ip-10-0-139-128 kubenswrapper[2562]: W0709 20:29:13.641172 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf8a0c21_1091_426b_b49b_eb0d52af8db2.slice/crio-5cc04f5d2e38bec22827a97774f068e6402fa9e685b56297942be66248ed97bd WatchSource:0}: Error finding container 5cc04f5d2e38bec22827a97774f068e6402fa9e685b56297942be66248ed97bd: Status 404 returned error can't find the container with id 5cc04f5d2e38bec22827a97774f068e6402fa9e685b56297942be66248ed97bd Jul 09 20:29:13.783433 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.783372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/1.log" Jul 09 20:29:13.788322 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.788302 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-svwdq_38e13fea-c6f3-4bec-bccd-9a976004b5bc/console-operator/2.log" Jul 09 20:29:13.828878 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.828848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" event={"ID":"df8a0c21-1091-426b-b49b-eb0d52af8db2","Type":"ContainerStarted","Data":"4e2c998d9668139d5278dfe45197f51e06a5a3556b9eeb055fcbad4dea465a4f"} Jul 09 20:29:13.828989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.828901 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" event={"ID":"df8a0c21-1091-426b-b49b-eb0d52af8db2","Type":"ContainerStarted","Data":"5cc04f5d2e38bec22827a97774f068e6402fa9e685b56297942be66248ed97bd"} Jul 09 20:29:13.828989 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.828967 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:13.846993 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:13.846948 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" podStartSLOduration=0.846935266 podStartE2EDuration="846.935266ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:29:13.845529123 +0000 UTC m=+2980.436289331" watchObservedRunningTime="2026-07-09 20:29:13.846935266 +0000 UTC m=+2980.437695471" Jul 09 20:29:14.865938 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:14.865908 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-njtmb_6b7a3330-d7dc-4363-b63b-93d50adc18dc/volume-data-source-validator/0.log" Jul 09 20:29:15.994216 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:15.994188 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w56d7_98146308-3725-4523-80ec-7d626c3aee7d/dns/0.log" Jul 09 20:29:16.016465 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:16.016440 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w56d7_98146308-3725-4523-80ec-7d626c3aee7d/kube-rbac-proxy/0.log" Jul 09 20:29:16.041028 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:16.040992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wtfh_2dd2e12b-47b3-42ed-8296-3ecea75e2d26/dns-node-resolver/0.log" Jul 09 20:29:16.649060 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:16.649036 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j984v_9e743b07-b293-4fb3-9e5e-982a16987054/node-ca/0.log" Jul 09 20:29:17.678922 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:17.678887 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vnbbh_e0a9fec9-fa38-4c3b-91d5-7d03a6026c97/istio-proxy/0.log" Jul 09 20:29:17.924856 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:17.924830 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5f4bc74d-x2hz8_24f55663-8dfb-43b4-a2e4-06fb45a5a38b/kube-auth-proxy/0.log" Jul 09 20:29:17.954061 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:17.953974 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/0.log" Jul 09 20:29:17.986711 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:17.986688 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/1.log" Jul 09 20:29:18.022204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:18.022177 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lch5w_3d6a813a-2aae-4908-9561-5f9b79fc8857/payload-pre-processing/0.log" Jul 09 20:29:18.052888 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:18.052865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7ptzq_2ee1edf3-ea29-47fb-be62-cc8260de47a1/payload-processing/0.log" Jul 09 20:29:18.842884 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:18.842854 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6qr9_71d87cb6-3fa7-4a7c-8cf8-678bae9c1cb2/serve-healthcheck-canary/0.log" Jul 09 20:29:19.340406 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:19.340377 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zf6f_df40c511-b6f9-4288-a237-c13f84d5e539/kube-rbac-proxy/0.log" Jul 09 20:29:19.362574 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:19.362550 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zf6f_df40c511-b6f9-4288-a237-c13f84d5e539/exporter/0.log" Jul 09 20:29:19.384714 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:19.384687 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6zf6f_df40c511-b6f9-4288-a237-c13f84d5e539/extractor/0.log" Jul 09 20:29:19.845204 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:19.845172 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l62zn/perf-node-gather-daemonset-zgptv" Jul 09 20:29:21.740545 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:21.740500 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4pn8q_e3db049e-97d4-4c00-a24f-ba5272bfc88f/manager/0.log" Jul 09 20:29:21.888431 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:21.888399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-khxfn_6f65dcb6-a83e-4aed-a5f0-abd60438e4f2/manager/0.log" Jul 09 20:29:21.991919 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:21.991840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6885c5f9d9-sx24t_d1ab6e2f-286f-40cc-860e-894515cd7a86/manager/0.log" Jul 09 20:29:22.013431 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:22.013407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g5fl4_5faa2b79-08e6-46e1-a4d8-94ab0068a98b/server/0.log" Jul 09 20:29:22.092766 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:22.092733 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ncn95_7cedd294-177f-46b8-996c-5150e85d805b/manager/0.log" Jul 09 20:29:22.173419 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:22.173390 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-776898495d-vxjqn_174887e9-7aaa-4a90-9538-fc9a29d000d4/manager/0.log" Jul 09 20:29:22.197437 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:22.197410 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-776898495d-vxjqn_174887e9-7aaa-4a90-9538-fc9a29d000d4/copy-manifests/0.log" Jul 09 20:29:23.700711 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:23.700681 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nxz8z_ece3c198-2336-4fbd-884d-0d33af9b0056/openshift-lws-operator/0.log" Jul 09 20:29:23.832425 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:23.832395 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vnbbh_e0a9fec9-fa38-4c3b-91d5-7d03a6026c97/istio-proxy/0.log" Jul 09 20:29:24.049818 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:24.049748 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5f4bc74d-x2hz8_24f55663-8dfb-43b4-a2e4-06fb45a5a38b/kube-auth-proxy/0.log" Jul 09 20:29:24.068368 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:24.068338 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/0.log" Jul 09 20:29:24.100822 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:24.100795 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/1.log" Jul 09 20:29:24.123995 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:24.123971 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lch5w_3d6a813a-2aae-4908-9561-5f9b79fc8857/payload-pre-processing/0.log" Jul 09 20:29:24.144734 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:24.144705 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7ptzq_2ee1edf3-ea29-47fb-be62-cc8260de47a1/payload-processing/0.log" Jul 09 20:29:25.335302 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.335271 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678vnbbh_e0a9fec9-fa38-4c3b-91d5-7d03a6026c97/istio-proxy/0.log" Jul 09 20:29:25.560801 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.560768 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5f4bc74d-x2hz8_24f55663-8dfb-43b4-a2e4-06fb45a5a38b/kube-auth-proxy/0.log" Jul 09 20:29:25.585764 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.585673 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/0.log" Jul 09 20:29:25.613924 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.613893 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-frw6q_50fb299f-3f49-4a3d-9631-99d70d8dc040/istio-proxy/1.log" Jul 09 20:29:25.640112 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.640087 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lch5w_3d6a813a-2aae-4908-9561-5f9b79fc8857/payload-pre-processing/0.log" Jul 09 20:29:25.663504 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:25.663476 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7ptzq_2ee1edf3-ea29-47fb-be62-cc8260de47a1/payload-processing/0.log" Jul 09 20:29:29.048362 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:29.048278 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-ch6g8_d488f561-09b6-48a7-af35-5805348a74f9/kube-storage-version-migrator-operator/1.log" Jul 09 20:29:29.049088 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:29.049072 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-ch6g8_d488f561-09b6-48a7-af35-5805348a74f9/kube-storage-version-migrator-operator/0.log" Jul 09 20:29:30.436095 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.436067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/kube-multus-additional-cni-plugins/0.log" Jul 09 20:29:30.457041 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.457016 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/egress-router-binary-copy/0.log" Jul 09 20:29:30.477910 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.477888 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/cni-plugins/0.log" Jul 09 20:29:30.500369 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.500346 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/bond-cni-plugin/0.log" Jul 09 20:29:30.521732 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.521709 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/routeoverride-cni/0.log" Jul 09 20:29:30.542655 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.542635 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/whereabouts-cni-bincopy/0.log" Jul 09 20:29:30.564379 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.564358 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrbsn_bbd07248-df9c-4edd-bd0d-847cc00c7fa1/whereabouts-cni/0.log" Jul 09 20:29:30.635001 ip-10-0-139-128 kubenswrapper[2562]: I0709 20:29:30.634969 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s8rsg_f5ee8c20-d161-49e4-ae02-87cb7e693f25/kube-multus/0.log"