Jul 22 01:43:14.294285 ip-10-0-133-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 01:43:14.294294 ip-10-0-133-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 01:43:14.294300 ip-10-0-133-122 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 01:43:14.294538 ip-10-0-133-122 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 01:43:24.543853 ip-10-0-133-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 01:43:24.543874 ip-10-0-133-122 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e753d493f3cc4fb69a9f0337e5f16279 -- Jul 22 01:45:21.996916 ip-10-0-133-122 systemd[1]: Starting Kubernetes Kubelet... Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.342960 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347785 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347803 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347809 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347813 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347817 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347822 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.437407 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347828 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.417150 ip-10-0-133-122 systemd[1]: Started Kubernetes Kubelet. Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347839 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347844 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347848 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347852 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347856 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347861 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347864 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347868 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347872 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347877 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347881 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347885 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347890 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347894 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347899 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347902 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347906 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347910 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347914 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.439707 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347917 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347921 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347925 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347929 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347933 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347936 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347940 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347943 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347948 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347952 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347956 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347960 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347965 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347968 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347972 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347976 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347980 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347983 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347987 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347991 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.440847 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347994 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.347999 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348003 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348006 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348011 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348015 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348019 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348023 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348027 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348030 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348034 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348038 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348041 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348045 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348049 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348052 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348056 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348060 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348063 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348067 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.441999 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348070 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348074 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348078 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348081 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348089 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348093 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348097 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348103 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348107 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348111 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348115 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348119 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348123 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348126 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348133 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348137 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348140 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348145 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348148 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.443177 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348153 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348156 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348160 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348164 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348168 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348172 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348176 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348180 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348183 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348187 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348191 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348197 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348202 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.348207 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349229 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349237 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349241 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349245 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349250 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.445094 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349255 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349259 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349263 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349267 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349271 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349275 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349278 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349282 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349286 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349290 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349293 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349298 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349301 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349305 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349309 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349313 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349316 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349320 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349324 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349329 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.446212 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349333 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349337 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349341 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349346 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349349 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349353 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349358 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349362 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349366 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349370 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349374 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349378 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349382 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349385 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349389 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349393 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349397 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349400 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349404 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.447544 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349409 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349414 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349418 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349424 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349430 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349434 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349439 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349443 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349447 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349451 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349455 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349459 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349463 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349467 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349471 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349474 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349478 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349482 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349486 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349490 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.448187 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349494 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349498 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349501 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349507 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349512 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349517 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349522 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349526 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349529 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349533 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349537 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349540 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349544 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349548 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349552 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349556 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349559 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349563 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349567 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349571 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.449918 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349574 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349577 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349581 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349585 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349589 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349592 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349597 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349601 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349605 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349608 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349612 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349616 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349619 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349623 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.349626 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350183 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350189 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350193 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350200 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350206 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.450619 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350210 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350215 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350219 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350223 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350227 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350232 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350236 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350240 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350244 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350248 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350253 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350257 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350260 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350264 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350268 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350272 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350276 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350280 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350284 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350287 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.451433 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350291 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350295 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350299 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350303 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350307 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350310 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350315 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350319 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350322 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350326 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350330 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350334 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350338 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350341 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350345 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350349 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350354 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350357 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350361 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350368 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.452007 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350373 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350377 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350382 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350386 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350390 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350393 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350397 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350400 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350404 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350408 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350411 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350414 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350418 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350422 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350425 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350428 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350433 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350437 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350445 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350449 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.452753 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350453 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350456 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350461 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350465 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350470 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350474 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350477 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350482 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350485 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350489 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350493 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350497 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350500 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350504 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350508 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350511 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350515 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350519 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350523 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.453334 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350527 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350531 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350535 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350538 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350543 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350546 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350550 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350554 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350558 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350562 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350566 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350572 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350575 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350579 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.350583 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350691 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350701 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350711 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350718 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350724 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350729 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 01:45:22.565452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350734 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350741 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350746 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350750 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350755 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350761 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350766 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350772 2585 flags.go:64] FLAG: --cgroup-root="" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350776 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350781 2585 flags.go:64] FLAG: --client-ca-file="" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350785 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350790 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350796 2585 flags.go:64] FLAG: --cluster-domain="" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350800 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350804 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350809 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350814 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350820 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350825 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350829 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350834 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350838 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350845 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350849 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 01:45:22.584059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350854 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350858 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350864 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350868 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350873 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350877 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350882 2585 flags.go:64] FLAG: --enable-server="true" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350886 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350893 2585 flags.go:64] FLAG: --event-burst="100" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350897 2585 flags.go:64] FLAG: --event-qps="50" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350902 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350906 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350910 2585 flags.go:64] FLAG: --eviction-hard="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350916 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350920 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350926 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350931 2585 flags.go:64] FLAG: --eviction-soft="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350935 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350939 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350944 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350948 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350952 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350957 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350961 2585 flags.go:64] FLAG: --feature-gates="" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350966 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 01:45:22.586566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350971 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350975 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350979 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350984 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350988 2585 flags.go:64] FLAG: --help="false" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350993 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.350998 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351002 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351006 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351011 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351016 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351021 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351025 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351029 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351034 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351038 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351044 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351048 2585 flags.go:64] FLAG: --kube-reserved="" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351052 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351057 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351061 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351065 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351070 2585 flags.go:64] FLAG: --lock-file="" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351075 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 01:45:22.587750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351079 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351083 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351091 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351095 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351100 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351104 2585 flags.go:64] FLAG: --logging-format="text" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351109 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351114 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351117 2585 flags.go:64] FLAG: --manifest-url="" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351121 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351127 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351132 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351138 2585 flags.go:64] FLAG: --max-pods="110" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351144 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351148 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351152 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351156 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351160 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351165 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351170 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351179 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351183 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351188 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 01:45:22.589622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351193 2585 flags.go:64] FLAG: --pod-cidr="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351197 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351205 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351209 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351213 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351217 2585 flags.go:64] FLAG: --port="10250" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351222 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351228 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d3163ce5eb4ae638" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351232 2585 flags.go:64] FLAG: --qos-reserved="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351237 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351241 2585 flags.go:64] FLAG: --register-node="true" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351245 2585 flags.go:64] FLAG: --register-with-taints="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351251 2585 flags.go:64] FLAG: --registry-burst="10" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351255 2585 flags.go:64] FLAG: --registry-qps="5" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351259 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351263 2585 flags.go:64] FLAG: --reserved-memory="" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351269 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351274 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351278 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351282 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351286 2585 flags.go:64] FLAG: --runonce="false" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351290 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351295 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351300 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351305 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 01:45:22.590619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351309 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351313 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351318 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351322 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351327 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351331 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351335 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351340 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351348 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351353 2585 flags.go:64] FLAG: --system-cgroups="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351357 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351364 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351368 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351372 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351379 2585 flags.go:64] FLAG: --tls-min-version="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351383 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351387 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351391 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351395 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351399 2585 flags.go:64] FLAG: --v="2" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351405 2585 flags.go:64] FLAG: --version="false" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351412 2585 flags.go:64] FLAG: --vmodule="" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351417 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.351422 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351594 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.593525 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351600 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351604 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351608 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351612 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351615 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351621 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351627 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351651 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351656 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351660 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351664 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351668 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351673 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351678 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351682 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351686 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351690 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351694 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351698 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.594898 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351702 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351706 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351712 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351715 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351719 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351723 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351727 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351731 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351735 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351739 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351742 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351746 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351750 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351753 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351757 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351761 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351764 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351768 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351774 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351778 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.597413 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351781 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351784 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351788 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351792 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351795 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351799 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351803 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351807 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351810 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351814 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351818 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351822 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351826 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351830 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351835 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351841 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351845 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351849 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351853 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.598422 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351857 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351861 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351865 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351869 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351872 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351876 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351880 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351884 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351887 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351891 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351896 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351901 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351905 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351909 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351912 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351916 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351920 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351923 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351927 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351931 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.600583 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351934 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351938 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351941 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351945 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351949 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351953 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351957 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351962 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351965 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351969 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351973 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351976 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351980 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351984 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351988 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351992 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351995 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.351999 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.352002 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.601587 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.352006 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.352014 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.360543 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.360561 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.360591 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.360602 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360681 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360688 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360691 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360694 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360698 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360701 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360704 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360707 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360710 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.603520 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360712 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360715 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360717 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360720 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360723 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360727 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360730 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360732 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360735 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360738 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360741 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360743 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360746 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360748 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360751 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360754 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360756 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360759 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360762 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360766 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.604305 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360769 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360771 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360777 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360780 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360783 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360786 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360789 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360792 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360795 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360798 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360801 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360803 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360806 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360809 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360811 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360814 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360817 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360821 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360823 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360826 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.606828 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360829 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360832 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360834 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360837 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360840 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360842 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360845 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360847 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360850 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360852 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360855 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360858 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360860 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360863 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360867 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360869 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360872 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360875 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360877 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360880 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.607651 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360882 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360886 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360890 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360893 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360896 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360898 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360903 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360906 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360909 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360915 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360918 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360921 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360924 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360927 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360929 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360932 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360934 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360937 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360939 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.610162 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360942 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360945 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360948 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360950 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360953 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360955 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360958 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360962 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360964 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360967 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.360970 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.360975 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361115 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361121 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361124 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:45:22.611132 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361127 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361130 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361133 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361136 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361138 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361141 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361144 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361147 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361150 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361153 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361156 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361158 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361161 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361163 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361166 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361168 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361171 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361175 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361179 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361182 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:45:22.613306 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361184 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361188 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361190 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361193 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361196 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361199 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361202 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361205 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361207 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361210 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361213 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361215 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361218 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361221 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361223 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361226 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361228 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361231 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361234 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361237 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:45:22.614119 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361240 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361242 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361246 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361249 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361251 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361254 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361256 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361259 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361261 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361264 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361266 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361269 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361271 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361274 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361276 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361279 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361281 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361285 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361288 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:45:22.616086 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361291 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361293 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361296 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361298 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361301 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361304 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361306 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361309 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361311 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361314 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361316 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361320 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361325 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361327 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361330 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361333 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361337 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361339 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361342 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:45:22.616874 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361345 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361347 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361350 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361352 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361355 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361357 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361360 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361362 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361365 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361367 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361370 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361374 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361376 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361379 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361381 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361385 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361389 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:45:22.765186 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:22.361392 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.361396 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.361512 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.363424 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.364474 2585 server.go:1019] "Starting client certificate rotation" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.364571 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.364609 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.366582 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.369621 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.382063 2585 log.go:25] "Validated CRI v1 runtime API" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.388035 2585 log.go:25] "Validated CRI v1 image API" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.389494 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 01:45:22.765962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.391911 2585 fs.go:135] Filesystem UUIDs: map[70b64e31-5f89-4822-8e66-1000077bd873:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cab169f0-bee6-462e-8c57-f38798f2c6ff:/dev/nvme0n1p4] Jul 22 01:45:22.769168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.391934 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 01:45:22.769168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.396240 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.397969 2585 manager.go:216] Machine: {Timestamp:2026-07-22 01:45:22.396718067 +0000 UTC m=+0.297596554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102948 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e1184c851eee2ef5d33b179972ce7 SystemUUID:ec2e1184-c851-eee2-ef5d-33b179972ce7 BootID:e753d493-f3cc-4fb6-9a9f-0337e5f16279 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1e:71:29:21:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1e:71:29:21:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:67:7b:f3:da:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.398086 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.398227 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.400758 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.400787 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.401354 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.401365 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.401372 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.401801 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.402863 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405524 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405673 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405700 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405709 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405725 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405734 2585 kubelet.go:398] "Adding apiserver pod source" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.405748 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 01:45:22.769574 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.406131 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.406145 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.407892 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.409495 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.409522 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410454 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410466 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410472 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410478 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410486 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410491 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410498 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410503 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410509 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410522 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410532 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.410537 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.411021 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.411028 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.416440 2585 server.go:1268] "Started kubelet" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.416882 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.418470 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.419332 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.419378 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.420547 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.420568 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.420682 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.420856 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.424691 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.424710 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.424941 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.425223 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.425291 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.425290 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.425310 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.425413 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.425419 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426390 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426407 2585 factory.go:55] Registering systemd factory Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426430 2585 factory.go:223] Registration of the systemd container factory successfully Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426816 2585 factory.go:153] Registering CRI-O factory Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426833 2585 factory.go:223] Registration of the crio container factory successfully Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426853 2585 factory.go:103] Registering Raw factory Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.426890 2585 manager.go:1196] Started watching for new ooms in manager Jul 22 01:45:22.772545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.427520 2585 manager.go:349] Starting recovery of all containers Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.426475 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-122.ec2.internal.18c4794ca3b259c9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-122.ec2.internal,UID:ip-10-0-133-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-122.ec2.internal,},FirstTimestamp:2026-07-22 01:45:22.416409033 +0000 UTC m=+0.317287518,LastTimestamp:2026-07-22 01:45:22.416409033 +0000 UTC m=+0.317287518,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-122.ec2.internal,}" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.431723 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.431810 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.436585 2585 manager.go:354] Recovery completed Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.443274 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445110 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445143 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445157 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445871 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445879 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.445894 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448026 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448133 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx8br" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448162 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx8br" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.448150 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-122.ec2.internal.18c4794ca56895c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-122.ec2.internal,UID:ip-10-0-133-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-122.ec2.internal,},FirstTimestamp:2026-07-22 01:45:22.445129153 +0000 UTC m=+0.346007644,LastTimestamp:2026-07-22 01:45:22.445129153 +0000 UTC m=+0.346007644,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-122.ec2.internal,}" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448816 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448839 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448848 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448960 2585 policy_none.go:49] "None policy: Start" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448977 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.448993 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.450103 2585 policy_none.go:47] "Start" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480324 2585 manager.go:337] "Starting Device Plugin manager" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.480356 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480365 2585 server.go:87] "Starting device plugin registration server" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480613 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480623 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480745 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480823 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.480832 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.481291 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 01:45:22.774344 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.481338 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.580832 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.581941 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.581971 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.581981 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.582007 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.589362 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.589383 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-122.ec2.internal\": node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.605810 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.669385 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.670535 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.670582 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.670602 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.670653 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.674236 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.706890 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.771245 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"] Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.771318 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.772822 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.772852 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.772863 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.774078 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.774181 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.774213 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775410 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775436 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775446 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775454 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775458 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.775976 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.775463 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.778440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.776753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.778440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.776775 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:45:22.778440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.778422 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:45:22.780620 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.778602 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:45:22.780620 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.778612 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.807996 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.815356 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-122.ec2.internal\" not found" node="ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.819517 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-122.ec2.internal\" not found" node="ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.828194 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.828217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.828235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a172972fbb39a4c6031f848e0d266ff-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"2a172972fbb39a4c6031f848e0d266ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:22.912582 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:22.908155 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928519 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a172972fbb39a4c6031f848e0d266ff-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"2a172972fbb39a4c6031f848e0d266ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928563 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7129788cc0ed725bc5853009237d25d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b7129788cc0ed725bc5853009237d25d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:22.928602 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a172972fbb39a4c6031f848e0d266ff-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"2a172972fbb39a4c6031f848e0d266ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.073198 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:23.008605 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.109008 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:23.108976 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.118138 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.118120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.121672 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.121607 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.209282 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:23.209251 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.309696 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:23.309667 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.364094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.364060 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 01:45:23.364257 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.364236 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 01:45:23.410447 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:23.410419 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found" Jul 22 01:45:23.451830 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.451794 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 01:40:22 +0000 UTC" deadline="2028-01-07 02:35:18.239817621 +0000 UTC" Jul 22 01:45:23.451830 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.451830 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12816h49m54.787991446s" Jul 22 01:45:23.507711 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.507626 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:45:23.525615 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.525583 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.537491 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.537469 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:45:23.539150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.539135 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" Jul 22 01:45:23.547351 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.547334 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:45:23.781549 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:23.781510 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7129788cc0ed725bc5853009237d25d.slice/crio-fc5fd978a7c0313228d9c5ef91059360052a464ad368626d7f8bfb83cdee058a WatchSource:0}: Error finding container fc5fd978a7c0313228d9c5ef91059360052a464ad368626d7f8bfb83cdee058a: Status 404 returned error can't find the container with id fc5fd978a7c0313228d9c5ef91059360052a464ad368626d7f8bfb83cdee058a Jul 22 01:45:23.781840 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:23.781822 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a172972fbb39a4c6031f848e0d266ff.slice/crio-d86b1d090d05d87c3de1bd2ed34e04b90e67fc89d7e135b8a76fdb5366d0cba8 WatchSource:0}: Error finding container d86b1d090d05d87c3de1bd2ed34e04b90e67fc89d7e135b8a76fdb5366d0cba8: Status 404 returned error can't find the container with id d86b1d090d05d87c3de1bd2ed34e04b90e67fc89d7e135b8a76fdb5366d0cba8 Jul 22 01:45:23.785687 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.785668 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:45:23.825831 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.825803 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:45:23.942989 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:23.942961 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:45:24.406961 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.406929 2585 apiserver.go:52] "Watching apiserver" Jul 22 01:45:24.414533 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.414509 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 01:45:24.417468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.417443 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-sgw7b","openshift-network-operator/iptables-alerter-wsxw7","openshift-ovn-kubernetes/ovnkube-node-j5w5c","kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal","openshift-cluster-node-tuning-operator/tuned-j4ctk","openshift-dns/node-resolver-wf4c5","openshift-image-registry/node-ca-xt2ch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal","openshift-network-diagnostics/network-check-target-6bqbj","kube-system/global-pull-secret-syncer-9xd9q","kube-system/konnectivity-agent-ftx5g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw","openshift-multus/multus-8w9bm","openshift-multus/multus-additional-cni-plugins-b544x"] Jul 22 01:45:24.419926 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.419903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.420024 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.419970 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:24.422509 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.422487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.424843 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.424799 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.425124 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.425108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.426417 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.426399 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 01:45:24.426626 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.426610 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pwlwk\"" Jul 22 01:45:24.427005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.426767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.427005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.426834 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 01:45:24.427005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.426919 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 01:45:24.427192 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.427158 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 01:45:24.427756 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.427363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.427756 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.427491 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 01:45:24.427756 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.427541 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 01:45:24.427927 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.427808 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcfkq\"" Jul 22 01:45:24.429469 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.429397 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.430625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.430603 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.430734 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.430671 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 01:45:24.430819 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.430792 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.431869 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.431829 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 01:45:24.431954 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.431943 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.432353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.432069 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 01:45:24.432353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.432162 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.432353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.432287 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v4brs\"" Jul 22 01:45:24.432487 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.432360 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mclr9\"" Jul 22 01:45:24.433996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.433591 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.435372 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435352 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:24.435457 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.435425 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:24.435745 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435724 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.435843 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-netns\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.435843 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435776 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-systemd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.435843 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-script-lib\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.435843 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435823 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-dbus\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-cnibin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-etc-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435907 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-netd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovn-node-metrics-cert\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-binary-copy\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.435965 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r7t\" (UniqueName: \"kubernetes.io/projected/8bb04989-996f-4024-a9c5-e68ee4e8fa90-kube-api-access-68r7t\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.436075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436021 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qqgh5\"" Jul 22 01:45:24.436468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436037 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-os-release\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436517 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436517 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-env-overrides\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436513 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwpw\" (UniqueName: \"kubernetes.io/projected/f3fea6a2-ead7-44f6-9105-cfe26b588298-kube-api-access-6fwpw\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-conf-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436574 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-slash\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436597 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-os-release\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436684 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-kubelet\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-multus-certs\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436751 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-etc-kubernetes\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-ovn\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436802 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-hostroot\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.436849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-node-log\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-system-cni-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436922 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cnibin\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-kubelet-config\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.436981 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-bin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-var-lib-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437037 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-log-socket\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437078 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-config\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437102 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-system-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437193 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-cni-binary-copy\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-netns\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437242 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-daemon-config\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437285 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-kubelet\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-bin\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-socket-dir-parent\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-k8s-cni-cncf-io\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-multus\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437442 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspct\" (UniqueName: \"kubernetes.io/projected/b48a44d6-960d-46f0-bab3-349611e8bfe5-kube-api-access-fspct\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-systemd-units\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.437873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.437520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.441440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.440935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.441555 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.439927 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.442796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.442229 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 01:45:24.442796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.442459 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.442796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.442649 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wbsj4\"" Jul 22 01:45:24.443231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.443213 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.443895 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.443879 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xmts\"" Jul 22 01:45:24.444614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.444352 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 01:45:24.445147 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.445123 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.445261 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.445241 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:45:24.445721 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.445586 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.445721 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.445700 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:24.447884 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.447744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.449980 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.449963 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.451771 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.451739 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-799wc\"" Jul 22 01:45:24.451771 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.451758 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 01:45:24.451931 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.451777 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 01:45:24.452470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.452025 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 01:45:24.452470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.452177 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 01:45:24.452796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.452780 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9th8c\"" Jul 22 01:45:24.453146 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.453005 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 01:45:24.461276 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.461254 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lbkgz" Jul 22 01:45:24.468055 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.468034 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lbkgz" Jul 22 01:45:24.518966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.518918 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:45:24.527322 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.527289 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 01:45:24.537841 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-os-release\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537853 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-lib-modules\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e139f97-4a64-40c6-aae3-a928b50bf79e-host\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537929 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-kubelet\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-ovn\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-os-release\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.537996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-node-log\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538037 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-ovn\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538053 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-run\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538083 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-host\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538123 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphw9\" (UniqueName: \"kubernetes.io/projected/8eabe7ab-9a59-4616-8637-b01ac6367c63-kube-api-access-jphw9\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-etc-selinux\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-hosts-file\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-kubelet-config\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538214 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-config\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538235 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.538320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538276 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-kubelet\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538370 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-kubelet-config\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e679e408-b43d-4ba2-9400-1dd133149d09-konnectivity-ca\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-node-log\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-tuned\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-cni-binary-copy\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538530 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-netns\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-daemon-config\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.538604 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-bin\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-netns\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-bin\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538719 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.538739 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:25.038721713 +0000 UTC m=+2.939600188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:24.539036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538867 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538904 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-socket-dir-parent\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538906 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-config\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspct\" (UniqueName: \"kubernetes.io/projected/b48a44d6-960d-46f0-bab3-349611e8bfe5-kube-api-access-fspct\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-systemd-units\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.538983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-systemd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539025 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-tmp\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539048 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-cni-binary-copy\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539070 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-etc-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-run-systemd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539094 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r7t\" (UniqueName: \"kubernetes.io/projected/8bb04989-996f-4024-a9c5-e68ee4e8fa90-kube-api-access-68r7t\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539122 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-daemon-config\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539133 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb44e2ea-c469-4333-a663-119d0e1d626d-host-slash\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-modprobe-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-socket-dir-parent\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539205 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-tmp-dir\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.539612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539239 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-systemd-units\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbn2\" (UniqueName: \"kubernetes.io/projected/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-kube-api-access-nwbn2\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-os-release\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-etc-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539353 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-os-release\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwpw\" (UniqueName: \"kubernetes.io/projected/f3fea6a2-ead7-44f6-9105-cfe26b588298-kube-api-access-6fwpw\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-device-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539418 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-conf\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539427 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-conf-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-slash\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539484 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-multus-conf-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-systemd\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539539 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-multus-certs\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-etc-kubernetes\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-slash\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539667 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cnibin\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539668 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-multus-certs\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-etc-kubernetes\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539722 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cnibin\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539734 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539754 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/eb44e2ea-c469-4333-a663-119d0e1d626d-kube-api-access-qz9b5\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-sys\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-var-lib-kubelet\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-hostroot\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-system-cni-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-bin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539929 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-var-lib-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539937 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bb04989-996f-4024-a9c5-e68ee4e8fa90-system-cni-dir\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539960 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-hostroot\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-log-socket\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.539990 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-var-lib-openvswitch\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.540962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540014 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-bin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-socket-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-system-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540077 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-log-socket\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-kubelet\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540122 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540148 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e139f97-4a64-40c6-aae3-a928b50bf79e-serviceca\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p5l\" (UniqueName: \"kubernetes.io/projected/1e139f97-4a64-40c6-aae3-a928b50bf79e-kube-api-access-52p5l\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-k8s-cni-cncf-io\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-multus\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-netns\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-script-lib\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-binary-copy\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb44e2ea-c469-4333-a663-119d0e1d626d-iptables-alerter-script\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-kubernetes\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540383 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-dbus\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.541722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-cnibin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-netd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovn-node-metrics-cert\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e679e408-b43d-4ba2-9400-1dd133149d09-agent-certs\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-registration-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540527 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-system-cni-dir\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540561 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-sys-fs\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysconfig\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540596 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-cni-netd\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-env-overrides\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540651 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-kubelet\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf28s\" (UniqueName: \"kubernetes.io/projected/de01926a-8602-45e6-a834-627fb6195a19-kube-api-access-mf28s\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540684 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-run-k8s-cni-cncf-io\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-host-var-lib-cni-multus\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.540734 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3fea6a2-ead7-44f6-9105-cfe26b588298-host-run-netns\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541053 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-env-overrides\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.542400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz2v\" (UniqueName: \"kubernetes.io/projected/ae9701f3-4150-4266-9621-5dff9bedb54f-kube-api-access-7vz2v\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.543105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541325 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 01:45:24.543105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e511869-a8c9-4c7f-8926-cff6e15d2849-dbus\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:24.543105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541412 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b48a44d6-960d-46f0-bab3-349611e8bfe5-cnibin\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.543105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.541847 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bb04989-996f-4024-a9c5-e68ee4e8fa90-cni-binary-copy\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.543105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.542003 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovnkube-script-lib\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.545850 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.545829 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3fea6a2-ead7-44f6-9105-cfe26b588298-ovn-node-metrics-cert\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.547253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.547232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwpw\" (UniqueName: \"kubernetes.io/projected/f3fea6a2-ead7-44f6-9105-cfe26b588298-kube-api-access-6fwpw\") pod \"ovnkube-node-j5w5c\" (UID: \"f3fea6a2-ead7-44f6-9105-cfe26b588298\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.547714 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.547621 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspct\" (UniqueName: \"kubernetes.io/projected/b48a44d6-960d-46f0-bab3-349611e8bfe5-kube-api-access-fspct\") pod \"multus-8w9bm\" (UID: \"b48a44d6-960d-46f0-bab3-349611e8bfe5\") " pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.548203 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.548175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r7t\" (UniqueName: \"kubernetes.io/projected/8bb04989-996f-4024-a9c5-e68ee4e8fa90-kube-api-access-68r7t\") pod \"multus-additional-cni-plugins-b544x\" (UID: \"8bb04989-996f-4024-a9c5-e68ee4e8fa90\") " pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.642202 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642169 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-run\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642202 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-host\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphw9\" (UniqueName: \"kubernetes.io/projected/8eabe7ab-9a59-4616-8637-b01ac6367c63-kube-api-access-jphw9\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-etc-selinux\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-hosts-file\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642285 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-host\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642290 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-run\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e679e408-b43d-4ba2-9400-1dd133149d09-konnectivity-ca\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-tuned\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642391 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642432 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-tmp\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb44e2ea-c469-4333-a663-119d0e1d626d-host-slash\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642473 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-modprobe-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-tmp-dir\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642510 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbn2\" (UniqueName: \"kubernetes.io/projected/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-kube-api-access-nwbn2\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-device-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-conf\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-systemd\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/eb44e2ea-c469-4333-a663-119d0e1d626d-kube-api-access-qz9b5\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-sys\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-etc-selinux\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642725 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-var-lib-kubelet\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-hosts-file\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.642783 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642682 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-var-lib-kubelet\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642784 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e679e408-b43d-4ba2-9400-1dd133149d09-konnectivity-ca\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-socket-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e139f97-4a64-40c6-aae3-a928b50bf79e-serviceca\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p5l\" (UniqueName: \"kubernetes.io/projected/1e139f97-4a64-40c6-aae3-a928b50bf79e-kube-api-access-52p5l\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb44e2ea-c469-4333-a663-119d0e1d626d-iptables-alerter-script\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-kubernetes\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e679e408-b43d-4ba2-9400-1dd133149d09-agent-certs\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.642999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-registration-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-sys-fs\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-tmp-dir\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysconfig\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf28s\" (UniqueName: \"kubernetes.io/projected/de01926a-8602-45e6-a834-627fb6195a19-kube-api-access-mf28s\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643083 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysconfig\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643109 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz2v\" (UniqueName: \"kubernetes.io/projected/ae9701f3-4150-4266-9621-5dff9bedb54f-kube-api-access-7vz2v\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-lib-modules\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e139f97-4a64-40c6-aae3-a928b50bf79e-host\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-socket-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643230 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e139f97-4a64-40c6-aae3-a928b50bf79e-host\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-sys\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643384 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-registration-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-kubernetes\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643420 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-sys-fs\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643562 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-sysctl-conf\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643686 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-modprobe-d\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.643915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.643698 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e139f97-4a64-40c6-aae3-a928b50bf79e-serviceca\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.644389 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.643921 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:24.644389 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.643980 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:25.143964457 +0000 UTC m=+3.044842931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:24.644389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.644345 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-lib-modules\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.644389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.644377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-systemd\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.644389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.644380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de01926a-8602-45e6-a834-627fb6195a19-device-dir\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.644625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.644505 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb44e2ea-c469-4333-a663-119d0e1d626d-host-slash\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.644760 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.644743 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb44e2ea-c469-4333-a663-119d0e1d626d-iptables-alerter-script\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.646165 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.646142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-etc-tuned\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.646399 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.646377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9701f3-4150-4266-9621-5dff9bedb54f-tmp\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.646744 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.646732 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e679e408-b43d-4ba2-9400-1dd133149d09-agent-certs\") pod \"konnectivity-agent-ftx5g\" (UID: \"e679e408-b43d-4ba2-9400-1dd133149d09\") " pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.648519 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.648490 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:24.648665 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.648646 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:24.648665 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.648669 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:24.648815 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:24.648738 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:25.148721381 +0000 UTC m=+3.049599869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:24.650803 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.650781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbn2\" (UniqueName: \"kubernetes.io/projected/2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58-kube-api-access-nwbn2\") pod \"node-resolver-wf4c5\" (UID: \"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58\") " pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.650910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.650882 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphw9\" (UniqueName: \"kubernetes.io/projected/8eabe7ab-9a59-4616-8637-b01ac6367c63-kube-api-access-jphw9\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:24.653984 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.653943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p5l\" (UniqueName: \"kubernetes.io/projected/1e139f97-4a64-40c6-aae3-a928b50bf79e-kube-api-access-52p5l\") pod \"node-ca-xt2ch\" (UID: \"1e139f97-4a64-40c6-aae3-a928b50bf79e\") " pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.654567 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.654538 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf28s\" (UniqueName: \"kubernetes.io/projected/de01926a-8602-45e6-a834-627fb6195a19-kube-api-access-mf28s\") pod \"aws-ebs-csi-driver-node-jt9zw\" (UID: \"de01926a-8602-45e6-a834-627fb6195a19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:24.654824 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.654797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/eb44e2ea-c469-4333-a663-119d0e1d626d-kube-api-access-qz9b5\") pod \"iptables-alerter-wsxw7\" (UID: \"eb44e2ea-c469-4333-a663-119d0e1d626d\") " pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.655118 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.655103 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz2v\" (UniqueName: \"kubernetes.io/projected/ae9701f3-4150-4266-9621-5dff9bedb54f-kube-api-access-7vz2v\") pod \"tuned-j4ctk\" (UID: \"ae9701f3-4150-4266-9621-5dff9bedb54f\") " pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.674697 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.674659 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" event={"ID":"2a172972fbb39a4c6031f848e0d266ff","Type":"ContainerStarted","Data":"d86b1d090d05d87c3de1bd2ed34e04b90e67fc89d7e135b8a76fdb5366d0cba8"} Jul 22 01:45:24.675664 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.675622 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b7129788cc0ed725bc5853009237d25d","Type":"ContainerStarted","Data":"fc5fd978a7c0313228d9c5ef91059360052a464ad368626d7f8bfb83cdee058a"} Jul 22 01:45:24.737998 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.737969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8w9bm" Jul 22 01:45:24.747841 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.747822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:24.757261 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.757242 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b544x" Jul 22 01:45:24.763797 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.763774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" Jul 22 01:45:24.771267 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.771248 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wf4c5" Jul 22 01:45:24.778700 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.778678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xt2ch" Jul 22 01:45:24.786178 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.786155 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wsxw7" Jul 22 01:45:24.793820 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.793798 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:24.799167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:24.799149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" Jul 22 01:45:25.046383 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.046304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:25.046536 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.046439 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:25.046536 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.046497 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:26.04648319 +0000 UTC m=+3.947361664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:25.146684 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.146656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:25.146846 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.146780 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:25.146846 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.146831 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:26.146817577 +0000 UTC m=+4.047696063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:25.247823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.247796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:25.247992 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.247973 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:25.248047 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.247998 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:25.248047 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.248010 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:25.248132 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.248070 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:26.248050288 +0000 UTC m=+4.148928780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:25.471470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.471424 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:40:24 +0000 UTC" deadline="2028-03-03 04:14:15.215265184 +0000 UTC" Jul 22 01:45:25.471470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.471465 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14162h28m49.743804014s" Jul 22 01:45:25.566722 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.566688 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb04989_996f_4024_a9c5_e68ee4e8fa90.slice/crio-81fdbb19dfa8cc620a2cad1416f2f6318d0f92aceef1e5a98347b64f072faa0d WatchSource:0}: Error finding container 81fdbb19dfa8cc620a2cad1416f2f6318d0f92aceef1e5a98347b64f072faa0d: Status 404 returned error can't find the container with id 81fdbb19dfa8cc620a2cad1416f2f6318d0f92aceef1e5a98347b64f072faa0d Jul 22 01:45:25.567759 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.567739 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9701f3_4150_4266_9621_5dff9bedb54f.slice/crio-ecf3953327409028fd4c2ad9efcece5f37afd756766c0853c7e44ed9c1b4b17f WatchSource:0}: Error finding container ecf3953327409028fd4c2ad9efcece5f37afd756766c0853c7e44ed9c1b4b17f: Status 404 returned error can't find the container with id ecf3953327409028fd4c2ad9efcece5f37afd756766c0853c7e44ed9c1b4b17f Jul 22 01:45:25.568745 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.568721 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e139f97_4a64_40c6_aae3_a928b50bf79e.slice/crio-ed601b1fe682390b31565fae5097da594fa571b895f6b65003a2febcdffb47e7 WatchSource:0}: Error finding container ed601b1fe682390b31565fae5097da594fa571b895f6b65003a2febcdffb47e7: Status 404 returned error can't find the container with id ed601b1fe682390b31565fae5097da594fa571b895f6b65003a2febcdffb47e7 Jul 22 01:45:25.571391 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.571371 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48a44d6_960d_46f0_bab3_349611e8bfe5.slice/crio-8009bfad7b0ba4787437ef5c29e2758b497ff129381652a81318399ea783b7ce WatchSource:0}: Error finding container 8009bfad7b0ba4787437ef5c29e2758b497ff129381652a81318399ea783b7ce: Status 404 returned error can't find the container with id 8009bfad7b0ba4787437ef5c29e2758b497ff129381652a81318399ea783b7ce Jul 22 01:45:25.572316 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.572286 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fea6a2_ead7_44f6_9105_cfe26b588298.slice/crio-e51a974caad3b7a10bb0b4585cd3abdf9566354b6795047b1e8579109294d1a3 WatchSource:0}: Error finding container e51a974caad3b7a10bb0b4585cd3abdf9566354b6795047b1e8579109294d1a3: Status 404 returned error can't find the container with id e51a974caad3b7a10bb0b4585cd3abdf9566354b6795047b1e8579109294d1a3 Jul 22 01:45:25.573112 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.573014 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb44e2ea_c469_4333_a663_119d0e1d626d.slice/crio-744d479f8334a4662e6be67746723fde312294bd6477321d475ff9ff54a06883 WatchSource:0}: Error finding container 744d479f8334a4662e6be67746723fde312294bd6477321d475ff9ff54a06883: Status 404 returned error can't find the container with id 744d479f8334a4662e6be67746723fde312294bd6477321d475ff9ff54a06883 Jul 22 01:45:25.573919 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:25.573814 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde01926a_8602_45e6_a834_627fb6195a19.slice/crio-f67ed20f503a4d391dd57a0ed04c57fed706cd442bfa38033673c4362a183e76 WatchSource:0}: Error finding container f67ed20f503a4d391dd57a0ed04c57fed706cd442bfa38033673c4362a183e76: Status 404 returned error can't find the container with id f67ed20f503a4d391dd57a0ed04c57fed706cd442bfa38033673c4362a183e76 Jul 22 01:45:25.671315 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.671278 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:25.671451 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.671414 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:25.671500 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.671456 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:25.671554 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:25.671537 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:25.679417 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.679385 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" event={"ID":"2a172972fbb39a4c6031f848e0d266ff","Type":"ContainerStarted","Data":"14e94aea06e1685cf05276c1a5e093a6aa6488261f172c16f4ab094e72ae2bb1"} Jul 22 01:45:25.680658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.680619 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftx5g" event={"ID":"e679e408-b43d-4ba2-9400-1dd133149d09","Type":"ContainerStarted","Data":"76e3ccf8e545fa96031d15926aa324248eaea00392e0eeb38bc125d3f7fa69b2"} Jul 22 01:45:25.682522 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.682495 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wf4c5" event={"ID":"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58","Type":"ContainerStarted","Data":"d6fb5a1efbfd44c2f947f5980bf1dd057d131a6046a8105b29787476b5fc76f3"} Jul 22 01:45:25.684167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.684114 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8w9bm" event={"ID":"b48a44d6-960d-46f0-bab3-349611e8bfe5","Type":"ContainerStarted","Data":"8009bfad7b0ba4787437ef5c29e2758b497ff129381652a81318399ea783b7ce"} Jul 22 01:45:25.685554 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.685533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xt2ch" event={"ID":"1e139f97-4a64-40c6-aae3-a928b50bf79e","Type":"ContainerStarted","Data":"ed601b1fe682390b31565fae5097da594fa571b895f6b65003a2febcdffb47e7"} Jul 22 01:45:25.686590 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.686567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" event={"ID":"ae9701f3-4150-4266-9621-5dff9bedb54f","Type":"ContainerStarted","Data":"ecf3953327409028fd4c2ad9efcece5f37afd756766c0853c7e44ed9c1b4b17f"} Jul 22 01:45:25.687622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.687605 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" event={"ID":"de01926a-8602-45e6-a834-627fb6195a19","Type":"ContainerStarted","Data":"f67ed20f503a4d391dd57a0ed04c57fed706cd442bfa38033673c4362a183e76"} Jul 22 01:45:25.688686 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.688664 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wsxw7" event={"ID":"eb44e2ea-c469-4333-a663-119d0e1d626d","Type":"ContainerStarted","Data":"744d479f8334a4662e6be67746723fde312294bd6477321d475ff9ff54a06883"} Jul 22 01:45:25.689776 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.689753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"e51a974caad3b7a10bb0b4585cd3abdf9566354b6795047b1e8579109294d1a3"} Jul 22 01:45:25.690731 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.690708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerStarted","Data":"81fdbb19dfa8cc620a2cad1416f2f6318d0f92aceef1e5a98347b64f072faa0d"} Jul 22 01:45:25.692263 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:25.692222 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" podStartSLOduration=2.692210663 podStartE2EDuration="2.692210663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:25.691736954 +0000 UTC m=+3.592615469" watchObservedRunningTime="2026-07-22 01:45:25.692210663 +0000 UTC m=+3.593089156" Jul 22 01:45:26.052660 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.052527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:26.052804 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.052723 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:26.052804 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.052783 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:28.052762899 +0000 UTC m=+5.953641374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:26.154164 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.154132 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:26.154741 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.154319 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:26.154741 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.154379 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:28.154361201 +0000 UTC m=+6.055239682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:26.255678 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.255023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:26.255678 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.255242 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:26.255678 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.255262 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:26.255678 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.255274 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:26.255678 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.255330 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:28.255313135 +0000 UTC m=+6.156191630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:26.472272 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.472211 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:40:24 +0000 UTC" deadline="2028-02-09 10:43:12.390142696 +0000 UTC" Jul 22 01:45:26.472272 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.472270 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13616h57m45.917876343s" Jul 22 01:45:26.673913 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.673883 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:26.674178 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:26.674014 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:26.720410 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.719138 2585 generic.go:356] "Generic (PLEG): container finished" podID="b7129788cc0ed725bc5853009237d25d" containerID="ab3fcc3568abb2b44663550b49e174162d7b1525557bffd0c0c7f6bb65c602ce" exitCode=0 Jul 22 01:45:26.720410 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:26.720135 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b7129788cc0ed725bc5853009237d25d","Type":"ContainerDied","Data":"ab3fcc3568abb2b44663550b49e174162d7b1525557bffd0c0c7f6bb65c602ce"} Jul 22 01:45:27.671473 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:27.671401 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:27.671955 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:27.671540 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:27.671955 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:27.671934 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:27.672079 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:27.672032 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:27.742645 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:27.742596 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b7129788cc0ed725bc5853009237d25d","Type":"ContainerStarted","Data":"6716196e88e757dd26e3083bbdd466d52dbf6e6d30ec08c84eb588ecaec42447"} Jul 22 01:45:28.070323 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:28.070237 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:28.070489 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.070451 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:28.070548 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.070514 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.070495213 +0000 UTC m=+9.971373696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:28.170819 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:28.170781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:28.171010 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.170975 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:28.171072 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.171035 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.171016829 +0000 UTC m=+10.071895305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:28.271312 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:28.271272 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:28.271481 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.271434 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:28.271481 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.271458 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:28.271481 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.271472 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:28.271732 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.271534 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.271516687 +0000 UTC m=+10.172395164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:28.675320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:28.675286 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:28.675797 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:28.675424 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:29.672361 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:29.671772 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:29.672361 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:29.671905 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:29.672361 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:29.671772 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:29.672604 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:29.672372 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:30.671502 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:30.671409 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:30.672062 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:30.671544 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:31.671886 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:31.671851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:31.672328 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:31.671886 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:31.672328 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:31.671994 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:31.672328 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:31.672097 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:32.103233 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:32.103136 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:32.103397 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.103308 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:32.103397 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.103382 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:40.103363483 +0000 UTC m=+18.004241980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:32.204284 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:32.204246 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:32.204487 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.204459 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:32.204555 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.204524 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:40.204506192 +0000 UTC m=+18.105384684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:32.305470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:32.305436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:32.305615 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.305564 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:32.305615 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.305580 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:32.305615 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.305589 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:32.305787 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.305658 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:40.305624695 +0000 UTC m=+18.206503174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:32.672559 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:32.672527 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:32.673036 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:32.672661 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:33.672123 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:33.671780 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:33.672123 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:33.671777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:33.672123 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:33.671930 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:33.672123 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:33.671987 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:34.674351 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:34.673865 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:34.674351 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:34.673990 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:35.671680 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:35.671649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:35.671846 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:35.671650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:35.671846 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:35.671763 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:35.671846 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:35.671833 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:36.671126 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:36.671091 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:36.671596 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:36.671215 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:37.671719 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:37.671670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:37.672194 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:37.671681 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:37.672194 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:37.671824 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:37.672194 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:37.671859 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:38.671481 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:38.671445 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:38.671712 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:38.671580 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:39.671737 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:39.671698 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:39.672183 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:39.671698 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:39.672183 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:39.671819 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:39.672183 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:39.671950 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:40.170559 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:40.170528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:40.170748 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.170726 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:40.170801 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.170789 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:56.170774333 +0000 UTC m=+34.071652811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:40.271775 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:40.271736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:40.271962 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.271901 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:40.272013 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.271975 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:56.271955931 +0000 UTC m=+34.172834411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:40.372798 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:40.372694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:40.372940 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.372900 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:40.372940 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.372935 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:40.373080 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.372950 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:40.373080 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.373010 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:45:56.372994297 +0000 UTC m=+34.273872776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:40.671712 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:40.671675 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:40.671868 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:40.671824 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:41.671770 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:41.671726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:41.671963 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:41.671777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:41.671963 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:41.671845 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:41.672369 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:41.671982 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:42.672023 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:42.671989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:42.672485 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:42.672102 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:43.671002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:43.670967 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:43.671002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:43.670985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:43.671243 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:43.671110 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:43.671243 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:43.671203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:44.671488 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.671068 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:44.672125 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:44.671602 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:44.778367 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.778337 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" event={"ID":"de01926a-8602-45e6-a834-627fb6195a19","Type":"ContainerStarted","Data":"1f100cb3404296648ed89652df5a04684b7cde5d018e4465b3b734e10c513b30"} Jul 22 01:45:44.780933 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.780906 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"a021b5deaa0dfe5f428e41a6d6093ab338fbf2c7d7d2d36e56d184cfb7b827db"} Jul 22 01:45:44.781075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.780970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"87aeabdcacd77097950dc39f3a7a7f1519b4b52cb536368f471801bb73aadec5"} Jul 22 01:45:44.781075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.781014 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"6a2e2185bb1bb46df968b417ca27ae8fa590c80c55e663c3a252677b0899fefd"} Jul 22 01:45:44.781075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.781049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"d839c96f6bc97bca991a67668c702067e539f3a5fd738059fadeb0bea8a7ad5a"} Jul 22 01:45:44.781075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.781077 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"21d5f05d93d244ece98bf6177d64486c9b62637f961e8c5e3a24a24aab75743e"} Jul 22 01:45:44.782318 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.782297 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="d15dc274c87f13e91a7a5f7d3f7df4d0777f4c4f52ae1389f9755060a5f5c4aa" exitCode=0 Jul 22 01:45:44.782407 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.782376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"d15dc274c87f13e91a7a5f7d3f7df4d0777f4c4f52ae1389f9755060a5f5c4aa"} Jul 22 01:45:44.783854 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.783801 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ftx5g" event={"ID":"e679e408-b43d-4ba2-9400-1dd133149d09","Type":"ContainerStarted","Data":"56a676f8de64805005dbf23af7a0fd8591173455a4386e7d65904332099ef901"} Jul 22 01:45:44.785187 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.785168 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wf4c5" event={"ID":"2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58","Type":"ContainerStarted","Data":"c862e3c2b11800e14c249131a00ad92c5ceb8b5b8a5e92d19d2f7a698445bd34"} Jul 22 01:45:44.786503 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.786486 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8w9bm" event={"ID":"b48a44d6-960d-46f0-bab3-349611e8bfe5","Type":"ContainerStarted","Data":"56fe5aff19e84bda9f1160dbfc790c80c998e0d970c8a0b3c475f3ab598cd814"} Jul 22 01:45:44.788543 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.788519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xt2ch" event={"ID":"1e139f97-4a64-40c6-aae3-a928b50bf79e","Type":"ContainerStarted","Data":"ec3d305e40974dbee77924e6fc8e0b251106f71a78185eaca673f32df6688872"} Jul 22 01:45:44.791385 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.791365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" event={"ID":"ae9701f3-4150-4266-9621-5dff9bedb54f","Type":"ContainerStarted","Data":"3d5327be954da5b1a3d0baeadd289561ebd196d4a58b213d54d108ff42efdb11"} Jul 22 01:45:44.810829 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.810779 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" podStartSLOduration=21.810763486 podStartE2EDuration="21.810763486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:27.763219913 +0000 UTC m=+5.664098426" watchObservedRunningTime="2026-07-22 01:45:44.810763486 +0000 UTC m=+22.711641982" Jul 22 01:45:44.824384 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.824339 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ftx5g" podStartSLOduration=4.257335211 podStartE2EDuration="22.824322064s" totalImagesPullingTime="18.566986853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.823783701 +0000 UTC m=+22.724662197" watchObservedRunningTime="2026-07-22 01:45:44.824322064 +0000 UTC m=+22.725200576" Jul 22 01:45:44.844154 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.844103 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8w9bm" podStartSLOduration=4.095488194 podStartE2EDuration="22.844086773s" totalImagesPullingTime="18.748598579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.844083682 +0000 UTC m=+22.744962199" watchObservedRunningTime="2026-07-22 01:45:44.844086773 +0000 UTC m=+22.744965270" Jul 22 01:45:44.865169 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.862600 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wf4c5" podStartSLOduration=4.250019768 podStartE2EDuration="22.862578756s" totalImagesPullingTime="18.612558988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.861623313 +0000 UTC m=+22.762501801" watchObservedRunningTime="2026-07-22 01:45:44.862578756 +0000 UTC m=+22.763457254" Jul 22 01:45:44.893598 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.893547 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j4ctk" podStartSLOduration=4.318513339 podStartE2EDuration="22.893528997s" totalImagesPullingTime="18.575015658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.893266411 +0000 UTC m=+22.794144929" watchObservedRunningTime="2026-07-22 01:45:44.893528997 +0000 UTC m=+22.794407493" Jul 22 01:45:44.893770 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:44.893688 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xt2ch" podStartSLOduration=4.319409429 podStartE2EDuration="22.893683458s" totalImagesPullingTime="18.574274029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.877690245 +0000 UTC m=+22.778568759" watchObservedRunningTime="2026-07-22 01:45:44.893683458 +0000 UTC m=+22.794561955" Jul 22 01:45:45.646742 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.646712 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 01:45:45.671188 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.671158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:45.671370 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.671158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:45.671370 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:45.671303 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:45.671465 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:45.671364 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:45.796266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.796171 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" event={"ID":"de01926a-8602-45e6-a834-627fb6195a19","Type":"ContainerStarted","Data":"efc5d6b473c5c81b2ba42a84e5dc8148d8d532cf3e1f9fd9038e45551c71bc9c"} Jul 22 01:45:45.797771 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.797739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wsxw7" event={"ID":"eb44e2ea-c469-4333-a663-119d0e1d626d","Type":"ContainerStarted","Data":"9dd1fb5596fffde903b13b96065edee231a32b5dd9d07bf894fdac18eb062d55"} Jul 22 01:45:45.801220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.801190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"a36fa52dc855fe401a89e65bc7a94e29e5ddee3010a9be3c70c905e57d62d452"} Jul 22 01:45:45.812823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:45.812769 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wsxw7" podStartSLOduration=5.222496738 podStartE2EDuration="23.812744583s" totalImagesPullingTime="18.590247845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:45.812302899 +0000 UTC m=+23.713181397" watchObservedRunningTime="2026-07-22 01:45:45.812744583 +0000 UTC m=+23.713623080" Jul 22 01:45:46.494367 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:46.494253 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T01:45:45.646736357Z","UUID":"69e0f6c1-5644-47e8-af19-dd63b567cb7b","Handler":null,"Name":"","Endpoint":""} Jul 22 01:45:46.496196 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:46.496147 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 01:45:46.496383 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:46.496367 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 01:45:46.671686 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:46.671626 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:46.671922 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:46.671776 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:47.671719 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:47.671678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:47.671719 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:47.671704 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:47.672372 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:47.671812 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:47.672372 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:47.671950 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:47.807951 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:47.807915 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" event={"ID":"de01926a-8602-45e6-a834-627fb6195a19","Type":"ContainerStarted","Data":"c3ba81052f187737e139462b89e11ee42984ee5226dbaf83629e1319a2a6d74e"} Jul 22 01:45:47.811570 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:47.811539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"22e4dc476150ab3dcf762a5c4f530b90175d927c99b4f0bda820402cf9b653fa"} Jul 22 01:45:47.826913 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:47.826849 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jt9zw" podStartSLOduration=4.938213743 podStartE2EDuration="25.826828162s" totalImagesPullingTime="20.888614419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:47.824988697 +0000 UTC m=+25.725867195" watchObservedRunningTime="2026-07-22 01:45:47.826828162 +0000 UTC m=+25.727706659" Jul 22 01:45:48.639819 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:48.639602 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:48.640301 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:48.640280 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:48.671402 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:48.671371 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:48.671590 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:48.671510 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:48.814594 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:48.814562 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:48.815058 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:48.814960 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ftx5g" Jul 22 01:45:49.671105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.671072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:49.671383 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.671073 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:49.671383 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:49.671180 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:49.671383 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:49.671274 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:49.819696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.819659 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" event={"ID":"f3fea6a2-ead7-44f6-9105-cfe26b588298","Type":"ContainerStarted","Data":"026b8bb95d3fd5cf694e692b4666285b7ddd9e05a83d7697c24d087d1f5a5968"} Jul 22 01:45:49.820601 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.819952 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:49.820601 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.820055 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:49.820601 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.820106 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:49.821753 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.821727 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="8bb140fb3d36411e9fd2b7cc326a2fd0477d1f8d288a7840f08fa8627468bbbf" exitCode=0 Jul 22 01:45:49.821865 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.821822 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"8bb140fb3d36411e9fd2b7cc326a2fd0477d1f8d288a7840f08fa8627468bbbf"} Jul 22 01:45:49.836467 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.836188 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:49.836659 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.836618 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:45:49.848175 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:49.848126 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" podStartSLOduration=9.216546598 podStartE2EDuration="27.848110978s" totalImagesPullingTime="18.63156438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:49.847163387 +0000 UTC m=+27.748041911" watchObservedRunningTime="2026-07-22 01:45:49.848110978 +0000 UTC m=+27.748989475" Jul 22 01:45:50.671607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:50.671576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:50.671812 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:50.671712 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:51.671034 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:51.671006 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:51.671442 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:51.671109 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:51.671442 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:51.671165 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:51.671442 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:51.671270 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:51.828984 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:51.828944 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="339bd94c0bb628c4420576a8781628f922fbd6b759f295c0c6c05aac7f23a3bf" exitCode=0 Jul 22 01:45:51.829153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:51.829036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"339bd94c0bb628c4420576a8781628f922fbd6b759f295c0c6c05aac7f23a3bf"} Jul 22 01:45:52.671816 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:52.671783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:52.672212 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:52.671868 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:52.704008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:52.703959 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:52.804740 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:52.804704 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:52.887760 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:52.887728 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:52.977374 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:52.977302 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:53.176093 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.176060 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:53.362247 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.362166 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:53.431481 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.431454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:53.557514 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.557483 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:53.671127 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.671104 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:53.671283 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:53.671214 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:53.671283 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.671268 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:53.671366 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:53.671350 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:53.836238 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.836203 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="e765d6b46b0e269cd1f7ffd867ef50fa77f6ef636c361a4438188038abe9f710" exitCode=0 Jul 22 01:45:53.836605 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.836263 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"e765d6b46b0e269cd1f7ffd867ef50fa77f6ef636c361a4438188038abe9f710"} Jul 22 01:45:53.958114 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:53.958033 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.035762 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.035726 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.296675 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.296577 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.392777 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.392734 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.486399 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.486364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.587499 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.587396 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.671473 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.671437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:54.671655 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:54.671591 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:54.686758 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.686727 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:54.898403 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:54.898319 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:55.196804 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.196763 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:55.255891 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.255857 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:55.473466 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.473378 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xd9q"] Jul 22 01:45:55.473607 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.473486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:55.473607 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:55.473582 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:55.476580 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.476399 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bqbj"] Jul 22 01:45:55.476580 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.476506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:55.476816 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:55.476608 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:55.477224 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.477201 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sgw7b"] Jul 22 01:45:55.477323 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:55.477305 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:55.477443 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:55.477413 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:56.194057 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:56.194016 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:56.194489 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.194150 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:56.194489 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.194226 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret podName:9e511869-a8c9-4c7f-8926-cff6e15d2849 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:28.194204598 +0000 UTC m=+66.095083089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret") pod "global-pull-secret-syncer-9xd9q" (UID: "9e511869-a8c9-4c7f-8926-cff6e15d2849") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:45:56.294589 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:56.294500 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:56.294763 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.294695 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:56.294820 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.294774 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs podName:8eabe7ab-9a59-4616-8637-b01ac6367c63 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:28.294751822 +0000 UTC m=+66.195630300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs") pod "network-metrics-daemon-sgw7b" (UID: "8eabe7ab-9a59-4616-8637-b01ac6367c63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:45:56.395645 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:56.395593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:56.395803 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.395767 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:45:56.395803 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.395792 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:45:56.395803 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.395803 2585 projected.go:196] Error preparing data for projected volume kube-api-access-6mzjv for pod openshift-network-diagnostics/network-check-target-6bqbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:56.395902 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.395857 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv podName:724b8752-2c6f-45d9-a545-7310096a80da nodeName:}" failed. No retries permitted until 2026-07-22 01:46:28.39584233 +0000 UTC m=+66.296720806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mzjv" (UniqueName: "kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv") pod "network-check-target-6bqbj" (UID: "724b8752-2c6f-45d9-a545-7310096a80da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:45:56.671918 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:56.671885 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:56.672102 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.672023 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:56.672102 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:56.672041 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:56.672212 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:56.672155 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:57.671566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:57.671531 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:57.672057 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:57.671680 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bqbj" podUID="724b8752-2c6f-45d9-a545-7310096a80da" Jul 22 01:45:57.812520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:57.812480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:57.912865 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:57.912828 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:57.975808 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:57.975625 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:58.338699 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:58.338596 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:58.671583 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:58.671550 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:45:58.672056 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:58.671728 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sgw7b" podUID="8eabe7ab-9a59-4616-8637-b01ac6367c63" Jul 22 01:45:58.672056 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:58.671770 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:45:58.672056 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:45:58.671856 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xd9q" podUID="9e511869-a8c9-4c7f-8926-cff6e15d2849" Jul 22 01:45:58.852148 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:58.852113 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="7042a520a363488d3c7b21aabbc88b31cdf4812f6686e3ff80d16578808142ec" exitCode=0 Jul 22 01:45:58.852319 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:58.852164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"7042a520a363488d3c7b21aabbc88b31cdf4812f6686e3ff80d16578808142ec"} Jul 22 01:45:59.290715 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.290615 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeReady" Jul 22 01:45:59.290884 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.290751 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 01:45:59.344695 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.344659 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bjtkx"] Jul 22 01:45:59.346435 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.346414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.347953 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.347927 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-prvz2"] Jul 22 01:45:59.349264 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.349242 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 01:45:59.349357 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.349244 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xrls9\"" Jul 22 01:45:59.349529 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.349514 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 01:45:59.349585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.349551 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 01:45:59.350043 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.350027 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.352687 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.352665 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 01:45:59.352799 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.352708 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldh2d\"" Jul 22 01:45:59.352946 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.352932 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 01:45:59.353018 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.352985 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 01:45:59.353675 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.353562 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 01:45:59.356241 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.356220 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bjtkx"] Jul 22 01:45:59.362285 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.362265 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-prvz2"] Jul 22 01:45:59.417308 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417261 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9332bbf-18e3-408e-8051-b2cc2b31ac69-crio-socket\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.417308 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417300 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059d8103-d9dc-47b5-9723-af409af96632-cert\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.417534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9332bbf-18e3-408e-8051-b2cc2b31ac69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.417534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417365 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h277r\" (UniqueName: \"kubernetes.io/projected/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-api-access-h277r\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.417534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.417534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9332bbf-18e3-408e-8051-b2cc2b31ac69-data-volume\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.417534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.417477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpcg\" (UniqueName: \"kubernetes.io/projected/059d8103-d9dc-47b5-9723-af409af96632-kube-api-access-pwpcg\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.445012 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.444983 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gnwr2"] Jul 22 01:45:59.447086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.447070 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.449793 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.449773 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 01:45:59.449979 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.449962 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz6sc\"" Jul 22 01:45:59.450077 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.450060 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 01:45:59.457679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.457625 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gnwr2"] Jul 22 01:45:59.518617 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518577 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184fc710-7c90-400b-b82a-3a91f60fcab2-tmp-dir\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.518822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518663 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9332bbf-18e3-408e-8051-b2cc2b31ac69-crio-socket\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.518822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059d8103-d9dc-47b5-9723-af409af96632-cert\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.518822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518728 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184fc710-7c90-400b-b82a-3a91f60fcab2-metrics-tls\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.518822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9332bbf-18e3-408e-8051-b2cc2b31ac69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.518822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518811 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h277r\" (UniqueName: \"kubernetes.io/projected/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-api-access-h277r\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184fc710-7c90-400b-b82a-3a91f60fcab2-config-volume\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518877 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9332bbf-18e3-408e-8051-b2cc2b31ac69-crio-socket\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518891 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518936 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9332bbf-18e3-408e-8051-b2cc2b31ac69-data-volume\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.518968 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92l4q\" (UniqueName: \"kubernetes.io/projected/184fc710-7c90-400b-b82a-3a91f60fcab2-kube-api-access-92l4q\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.519040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.519005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpcg\" (UniqueName: \"kubernetes.io/projected/059d8103-d9dc-47b5-9723-af409af96632-kube-api-access-pwpcg\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.519421 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.519389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9332bbf-18e3-408e-8051-b2cc2b31ac69-data-volume\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.519765 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.519746 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.522870 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.522848 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9332bbf-18e3-408e-8051-b2cc2b31ac69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.522977 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.522894 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/059d8103-d9dc-47b5-9723-af409af96632-cert\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.528401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.528379 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpcg\" (UniqueName: \"kubernetes.io/projected/059d8103-d9dc-47b5-9723-af409af96632-kube-api-access-pwpcg\") pod \"ingress-canary-bjtkx\" (UID: \"059d8103-d9dc-47b5-9723-af409af96632\") " pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.528513 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.528468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h277r\" (UniqueName: \"kubernetes.io/projected/b9332bbf-18e3-408e-8051-b2cc2b31ac69-kube-api-access-h277r\") pod \"insights-runtime-extractor-prvz2\" (UID: \"b9332bbf-18e3-408e-8051-b2cc2b31ac69\") " pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.552698 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.552614 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:45:59.619705 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.619675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184fc710-7c90-400b-b82a-3a91f60fcab2-metrics-tls\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.619821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.619723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184fc710-7c90-400b-b82a-3a91f60fcab2-config-volume\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.619821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.619761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92l4q\" (UniqueName: \"kubernetes.io/projected/184fc710-7c90-400b-b82a-3a91f60fcab2-kube-api-access-92l4q\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.619821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.619812 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184fc710-7c90-400b-b82a-3a91f60fcab2-tmp-dir\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.620208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.620185 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184fc710-7c90-400b-b82a-3a91f60fcab2-tmp-dir\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.620427 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.620411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/184fc710-7c90-400b-b82a-3a91f60fcab2-config-volume\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.622189 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.622167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/184fc710-7c90-400b-b82a-3a91f60fcab2-metrics-tls\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.628468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.628442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92l4q\" (UniqueName: \"kubernetes.io/projected/184fc710-7c90-400b-b82a-3a91f60fcab2-kube-api-access-92l4q\") pod \"dns-default-gnwr2\" (UID: \"184fc710-7c90-400b-b82a-3a91f60fcab2\") " pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.657568 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.657542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bjtkx" Jul 22 01:45:59.663199 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.663179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-prvz2" Jul 22 01:45:59.671130 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.671109 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:45:59.674013 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.673983 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:45:59.674560 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.674115 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wgwz\"" Jul 22 01:45:59.674560 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.674120 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:45:59.756440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.756156 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gnwr2" Jul 22 01:45:59.817507 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.817268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-prvz2"] Jul 22 01:45:59.823294 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:59.823135 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9332bbf_18e3_408e_8051_b2cc2b31ac69.slice/crio-6f1cf9b50881f9b1ddfedf55162568b7f1aecd841b9d791c9db215b4940e8e61 WatchSource:0}: Error finding container 6f1cf9b50881f9b1ddfedf55162568b7f1aecd841b9d791c9db215b4940e8e61: Status 404 returned error can't find the container with id 6f1cf9b50881f9b1ddfedf55162568b7f1aecd841b9d791c9db215b4940e8e61 Jul 22 01:45:59.823906 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.823657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bjtkx"] Jul 22 01:45:59.826420 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:59.826354 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059d8103_d9dc_47b5_9723_af409af96632.slice/crio-dd44c1965496f63b7cf3b50859701af4eaf2c30fcec4078951f5ea1bfb68f773 WatchSource:0}: Error finding container dd44c1965496f63b7cf3b50859701af4eaf2c30fcec4078951f5ea1bfb68f773: Status 404 returned error can't find the container with id dd44c1965496f63b7cf3b50859701af4eaf2c30fcec4078951f5ea1bfb68f773 Jul 22 01:45:59.859220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.859187 2585 generic.go:356] "Generic (PLEG): container finished" podID="8bb04989-996f-4024-a9c5-e68ee4e8fa90" containerID="39e17a3146dfd724eb8122645bc0babf3b071d59836a0a59cf6d9f3d707569c4" exitCode=0 Jul 22 01:45:59.859385 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.859333 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerDied","Data":"39e17a3146dfd724eb8122645bc0babf3b071d59836a0a59cf6d9f3d707569c4"} Jul 22 01:45:59.866596 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.863296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bjtkx" event={"ID":"059d8103-d9dc-47b5-9723-af409af96632","Type":"ContainerStarted","Data":"dd44c1965496f63b7cf3b50859701af4eaf2c30fcec4078951f5ea1bfb68f773"} Jul 22 01:45:59.866596 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.865896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-prvz2" event={"ID":"b9332bbf-18e3-408e-8051-b2cc2b31ac69","Type":"ContainerStarted","Data":"6f1cf9b50881f9b1ddfedf55162568b7f1aecd841b9d791c9db215b4940e8e61"} Jul 22 01:45:59.908713 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:45:59.908685 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gnwr2"] Jul 22 01:45:59.913149 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:45:59.913124 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184fc710_7c90_400b_b82a_3a91f60fcab2.slice/crio-538b78be48fe653ef8ef48669472b4a16c3e620b2ed8b9a143138abe7345280a WatchSource:0}: Error finding container 538b78be48fe653ef8ef48669472b4a16c3e620b2ed8b9a143138abe7345280a: Status 404 returned error can't find the container with id 538b78be48fe653ef8ef48669472b4a16c3e620b2ed8b9a143138abe7345280a Jul 22 01:46:00.584480 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.584404 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:00.651212 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.651129 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:00.672157 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.671357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:46:00.672839 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.672501 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:46:00.675915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.675673 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cgfms\"" Jul 22 01:46:00.675915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.675696 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:46:00.675915 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.675708 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:46:00.876315 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.875491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b544x" event={"ID":"8bb04989-996f-4024-a9c5-e68ee4e8fa90","Type":"ContainerStarted","Data":"6f4516d45c7dd6801dad4cecc67f15541cc74eb6734d57f6060330775d2bd98f"} Jul 22 01:46:00.878193 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.878104 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gnwr2" event={"ID":"184fc710-7c90-400b-b82a-3a91f60fcab2","Type":"ContainerStarted","Data":"538b78be48fe653ef8ef48669472b4a16c3e620b2ed8b9a143138abe7345280a"} Jul 22 01:46:00.880027 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.879948 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:00.900060 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.900000 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b544x" podStartSLOduration=9.453033214 podStartE2EDuration="38.899975348s" totalImagesPullingTime="29.446942134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:00.898265568 +0000 UTC m=+38.799144066" watchObservedRunningTime="2026-07-22 01:46:00.899975348 +0000 UTC m=+38.800853846" Jul 22 01:46:00.981816 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:00.981781 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.085707 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.085658 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.181978 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.181897 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.432585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.432506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.516140 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.516105 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.575517 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.575478 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.806139 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.806022 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:01.828470 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.828442 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:01.828655 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.828579 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.831433 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.831407 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 01:46:01.831586 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.831527 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 01:46:01.831676 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.831664 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h58b6\"" Jul 22 01:46:01.832601 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.832547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 01:46:01.832601 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.832580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 01:46:01.832800 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.832675 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 01:46:01.832800 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.832791 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 01:46:01.832900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.832853 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 01:46:01.929939 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.929904 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:01.939040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939009 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.939219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.939219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939086 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.939219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939151 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwlm\" (UniqueName: \"kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.939219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:01.939219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:01.939214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.029611 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.029570 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:02.039916 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.039881 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwlm\" (UniqueName: \"kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.039916 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.039923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.040162 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.039958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.040162 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.040119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.040249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.040170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.040249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.040206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.041233 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.041180 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.041233 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.041191 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.041417 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.041317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.044700 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.044670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.044856 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.044833 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.049135 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.049082 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwlm\" (UniqueName: \"kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm\") pod \"console-6f4cc4b4b5-n6ntm\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:02.141616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:02.141297 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:03.487541 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:03.487497 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:03.545204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:03.545164 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:03.940626 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:03.940584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:03.974081 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:03.974044 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rm88f"] Jul 22 01:46:04.039418 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.039375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:04.094372 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.094345 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rm88f"] Jul 22 01:46:04.094542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.094452 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.096973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.096924 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 01:46:04.097119 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.097027 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 01:46:04.097188 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.097114 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-746n6\"" Jul 22 01:46:04.097188 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.097142 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 01:46:04.098418 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.098320 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 01:46:04.098418 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.098329 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 01:46:04.260940 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.260851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c09086b7-d3dc-49ec-b888-ecf2929a580e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.261090 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.260987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79kj\" (UniqueName: \"kubernetes.io/projected/c09086b7-d3dc-49ec-b888-ecf2929a580e-kube-api-access-z79kj\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.261090 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.261057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.261215 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.261116 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.362113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.362075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.362295 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.362140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c09086b7-d3dc-49ec-b888-ecf2929a580e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.362295 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.362202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79kj\" (UniqueName: \"kubernetes.io/projected/c09086b7-d3dc-49ec-b888-ecf2929a580e-kube-api-access-z79kj\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.362295 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.362239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.362446 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:04.362372 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 22 01:46:04.362498 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:04.362456 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls podName:c09086b7-d3dc-49ec-b888-ecf2929a580e nodeName:}" failed. No retries permitted until 2026-07-22 01:46:04.862422588 +0000 UTC m=+42.763301064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-rm88f" (UID: "c09086b7-d3dc-49ec-b888-ecf2929a580e") : secret "prometheus-operator-tls" not found Jul 22 01:46:04.362845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.362822 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c09086b7-d3dc-49ec-b888-ecf2929a580e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.365003 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.364981 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.376778 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.376753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79kj\" (UniqueName: \"kubernetes.io/projected/c09086b7-d3dc-49ec-b888-ecf2929a580e-kube-api-access-z79kj\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.536535 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.536453 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:04.866622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.866575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:04.869442 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:04.869416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c09086b7-d3dc-49ec-b888-ecf2929a580e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rm88f\" (UID: \"c09086b7-d3dc-49ec-b888-ecf2929a580e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:05.004288 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:05.004258 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" Jul 22 01:46:06.345398 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.345363 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:06.631292 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:06.631208 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45141ee_5c79_418c_b252_b6ca59be6cb3.slice/crio-3eba6da27d02eeb0671a4b89dbfc8d4480d435a69e2a6e49d4c5a285b6906df0 WatchSource:0}: Error finding container 3eba6da27d02eeb0671a4b89dbfc8d4480d435a69e2a6e49d4c5a285b6906df0: Status 404 returned error can't find the container with id 3eba6da27d02eeb0671a4b89dbfc8d4480d435a69e2a6e49d4c5a285b6906df0 Jul 22 01:46:06.767309 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.767284 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rm88f"] Jul 22 01:46:06.771127 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:06.771097 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09086b7_d3dc_49ec_b888_ecf2929a580e.slice/crio-3bf5385dd30893c7b4086c0085542a41f495cb85f4f477ff66bd0c01deb56c0a WatchSource:0}: Error finding container 3bf5385dd30893c7b4086c0085542a41f495cb85f4f477ff66bd0c01deb56c0a: Status 404 returned error can't find the container with id 3bf5385dd30893c7b4086c0085542a41f495cb85f4f477ff66bd0c01deb56c0a Jul 22 01:46:06.820998 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.820949 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:06.898129 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.898066 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4cc4b4b5-n6ntm" event={"ID":"b45141ee-5c79-418c-b252-b6ca59be6cb3","Type":"ContainerStarted","Data":"3eba6da27d02eeb0671a4b89dbfc8d4480d435a69e2a6e49d4c5a285b6906df0"} Jul 22 01:46:06.899851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.899825 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bjtkx" event={"ID":"059d8103-d9dc-47b5-9723-af409af96632","Type":"ContainerStarted","Data":"bbe80f4c1117990faf9782dbd0ffd7e209f72e7eb9b05e2a2bdf66fc2fc639c3"} Jul 22 01:46:06.901529 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.901503 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gnwr2" event={"ID":"184fc710-7c90-400b-b82a-3a91f60fcab2","Type":"ContainerStarted","Data":"e4310a9c0b012776c64c9387eda2c190aa0d52f6575871160bb2b9dc1e095c35"} Jul 22 01:46:06.902826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.902800 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" event={"ID":"c09086b7-d3dc-49ec-b888-ecf2929a580e","Type":"ContainerStarted","Data":"3bf5385dd30893c7b4086c0085542a41f495cb85f4f477ff66bd0c01deb56c0a"} Jul 22 01:46:06.909969 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.909947 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:06.938816 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:06.938362 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bjtkx" podStartSLOduration=1.138806762 podStartE2EDuration="7.938339046s" totalImagesPullingTime="6.799532284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:06.937500143 +0000 UTC m=+44.838378654" watchObservedRunningTime="2026-07-22 01:46:06.938339046 +0000 UTC m=+44.839217540" Jul 22 01:46:07.797237 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:07.797201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:07.896888 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:07.896815 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:07.907560 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:07.907518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-prvz2" event={"ID":"b9332bbf-18e3-408e-8051-b2cc2b31ac69","Type":"ContainerStarted","Data":"824e91546e3f36dd0e502fdced4872eacbdd924bda786962181dbbd345035a20"} Jul 22 01:46:07.909786 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:07.909749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gnwr2" event={"ID":"184fc710-7c90-400b-b82a-3a91f60fcab2","Type":"ContainerStarted","Data":"a3314aec32108390a0c33ed984371a5aedd7a6fd054b0ddb1574533d57978788"} Jul 22 01:46:07.929919 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:07.929856 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gnwr2" podStartSLOduration=2.647059527 podStartE2EDuration="8.929832646s" totalImagesPullingTime="6.282773119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:07.92971044 +0000 UTC m=+45.830588940" watchObservedRunningTime="2026-07-22 01:46:07.929832646 +0000 UTC m=+45.830711147" Jul 22 01:46:08.916249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:08.916011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" event={"ID":"c09086b7-d3dc-49ec-b888-ecf2929a580e","Type":"ContainerStarted","Data":"2ea9a944959dc558964b048b359c6e9d6bfd67ab0d55be931f9eaf89c1665db9"} Jul 22 01:46:08.917486 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:08.917463 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gnwr2" Jul 22 01:46:09.122033 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.121945 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:09.382400 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.382319 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:46:09.395016 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.394984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.395561 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.395532 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:46:09.405061 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.405035 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 01:46:09.506076 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tgc\" (UniqueName: \"kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.506493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.506313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.607621 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607578 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.607826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.607826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.607826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607743 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.607826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.608014 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607841 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.608014 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.607870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tgc\" (UniqueName: \"kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.609058 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.608469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.609058 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.608773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.609058 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.608986 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.609307 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.609107 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.610874 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.610849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.611358 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.611341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.620649 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.620606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tgc\" (UniqueName: \"kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc\") pod \"console-65769658cf-ht7s9\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.706993 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.706950 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:09.972395 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:09.972311 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:10.071684 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:10.071651 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:10.191113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:10.191074 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:10.245182 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:10.245092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:12.658944 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.658870 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:46:12.662575 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:12.662545 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc239d27a_4d19_49f0_af31_6a344ed2e14b.slice/crio-e867ff2d904d2cee97cb5a340902cc12c1638af0ad783fd3b3884d16af8b4422 WatchSource:0}: Error finding container e867ff2d904d2cee97cb5a340902cc12c1638af0ad783fd3b3884d16af8b4422: Status 404 returned error can't find the container with id e867ff2d904d2cee97cb5a340902cc12c1638af0ad783fd3b3884d16af8b4422 Jul 22 01:46:12.930138 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.930101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4cc4b4b5-n6ntm" event={"ID":"b45141ee-5c79-418c-b252-b6ca59be6cb3","Type":"ContainerStarted","Data":"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352"} Jul 22 01:46:12.931692 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.931666 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65769658cf-ht7s9" event={"ID":"c239d27a-4d19-49f0-af31-6a344ed2e14b","Type":"ContainerStarted","Data":"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285"} Jul 22 01:46:12.931818 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.931709 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65769658cf-ht7s9" event={"ID":"c239d27a-4d19-49f0-af31-6a344ed2e14b","Type":"ContainerStarted","Data":"e867ff2d904d2cee97cb5a340902cc12c1638af0ad783fd3b3884d16af8b4422"} Jul 22 01:46:12.933338 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.933314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-prvz2" event={"ID":"b9332bbf-18e3-408e-8051-b2cc2b31ac69","Type":"ContainerStarted","Data":"15ce87629791dd842b0d2540a28e029354ad76c2f8edb99cb185fb0f6b30558b"} Jul 22 01:46:12.934986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.934965 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" event={"ID":"c09086b7-d3dc-49ec-b888-ecf2929a580e","Type":"ContainerStarted","Data":"59764f2d188fabaf381a6e53877afc4d46804a094009f00fe2d25c191695d90c"} Jul 22 01:46:12.947386 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.947332 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f4cc4b4b5-n6ntm" podStartSLOduration=6.058411287 podStartE2EDuration="11.947317434s" totalImagesPullingTime="5.888906147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:12.946088335 +0000 UTC m=+50.846966850" watchObservedRunningTime="2026-07-22 01:46:12.947317434 +0000 UTC m=+50.848195909" Jul 22 01:46:12.962043 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.961991 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rm88f" podStartSLOduration=8.042895678 podStartE2EDuration="9.961974367s" totalImagesPullingTime="1.919078689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:12.961426253 +0000 UTC m=+50.862304760" watchObservedRunningTime="2026-07-22 01:46:12.961974367 +0000 UTC m=+50.862852863" Jul 22 01:46:12.977910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:12.977857 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65769658cf-ht7s9" podStartSLOduration=3.977840846 podStartE2EDuration="3.977840846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:12.977545622 +0000 UTC m=+50.878424119" watchObservedRunningTime="2026-07-22 01:46:12.977840846 +0000 UTC m=+50.878719343" Jul 22 01:46:13.691203 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:13.691163 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:13.795057 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:13.795020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:14.919630 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:14.919590 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.022660 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.022616 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.120132 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.120080 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.282271 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.282187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.306512 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.306479 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw"] Jul 22 01:46:15.382166 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.382128 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.408705 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.408658 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw"] Jul 22 01:46:15.408705 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.408712 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzjv7"] Jul 22 01:46:15.408951 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.408775 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.411501 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.411477 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 01:46:15.411681 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.411476 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 01:46:15.411681 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.411591 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8trv6\"" Jul 22 01:46:15.428762 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.428724 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzjv7"] Jul 22 01:46:15.428928 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.428790 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bhrsk"] Jul 22 01:46:15.428986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.428968 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.431427 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.431405 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 01:46:15.431652 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.431602 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 01:46:15.431779 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.431690 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bxn57\"" Jul 22 01:46:15.431962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.431943 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 01:46:15.444610 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.444582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.448493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.448263 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 01:46:15.448493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.448335 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p24dd\"" Jul 22 01:46:15.448493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.448382 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 01:46:15.448493 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.448413 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 01:46:15.449733 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.449678 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.453235 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453355 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-sys\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453355 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453259 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453355 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453332 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-wtmp\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8k8\" (UniqueName: \"kubernetes.io/projected/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-api-access-db8k8\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.453515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453409 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64ns\" (UniqueName: \"kubernetes.io/projected/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-kube-api-access-q64ns\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.453515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453502 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cf39828-cdf6-495e-96ee-b110e01dce91-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453569 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453598 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-metrics-client-ca\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qrv\" (UniqueName: \"kubernetes.io/projected/1558c0c0-4b52-4771-b01a-79c8786560c6-kube-api-access-49qrv\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.453726 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.454019 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.454019 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.454019 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.454019 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-root\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.454019 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.453865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-textfile\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554358 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-textfile\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554358 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-sys\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-wtmp\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8k8\" (UniqueName: \"kubernetes.io/projected/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-api-access-db8k8\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554510 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554553 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64ns\" (UniqueName: \"kubernetes.io/projected/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-kube-api-access-q64ns\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cf39828-cdf6-495e-96ee-b110e01dce91-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.554679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554666 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-metrics-client-ca\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qrv\" (UniqueName: \"kubernetes.io/projected/1558c0c0-4b52-4771-b01a-79c8786560c6-kube-api-access-49qrv\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554738 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554776 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-root\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.555113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.554991 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-root\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.555526 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.555243 2585 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 01:46:15.555526 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.555309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-textfile\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.555526 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.555330 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls podName:3cf39828-cdf6-495e-96ee-b110e01dce91 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:16.055294802 +0000 UTC m=+53.956173318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-nzjv7" (UID: "3cf39828-cdf6-495e-96ee-b110e01dce91") : secret "kube-state-metrics-tls" not found Jul 22 01:46:15.556171 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.556144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.556276 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.556190 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.556789 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.556756 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1558c0c0-4b52-4771-b01a-79c8786560c6-metrics-client-ca\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.557160 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.557134 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 01:46:15.557260 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.557194 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls podName:1558c0c0-4b52-4771-b01a-79c8786560c6 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:16.057173672 +0000 UTC m=+53.958052149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls") pod "node-exporter-bhrsk" (UID: "1558c0c0-4b52-4771-b01a-79c8786560c6") : secret "node-exporter-tls" not found Jul 22 01:46:15.557260 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.557242 2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 01:46:15.557376 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:15.557302 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls podName:ac42c982-7b05-4ba5-a82a-b1e9a1668eb1 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:16.057286386 +0000 UTC m=+53.958164863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-ck5dw" (UID: "ac42c982-7b05-4ba5-a82a-b1e9a1668eb1") : secret "openshift-state-metrics-tls" not found Jul 22 01:46:15.557554 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.557131 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.557621 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.557567 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf39828-cdf6-495e-96ee-b110e01dce91-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.557694 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.557675 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-wtmp\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.559798 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.557732 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1558c0c0-4b52-4771-b01a-79c8786560c6-sys\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.559798 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.559742 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cf39828-cdf6-495e-96ee-b110e01dce91-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.560329 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.560300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.563694 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.562028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.563694 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.562134 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.566916 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.566812 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.568302 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.568280 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8k8\" (UniqueName: \"kubernetes.io/projected/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-api-access-db8k8\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:15.569565 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.568683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qrv\" (UniqueName: \"kubernetes.io/projected/1558c0c0-4b52-4771-b01a-79c8786560c6-kube-api-access-49qrv\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:15.570370 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.570350 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64ns\" (UniqueName: \"kubernetes.io/projected/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-kube-api-access-q64ns\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:15.664573 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.664538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.764749 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.764707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.819450 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.819367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.875709 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.875668 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:15.927990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:15.927951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.058887 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.058854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:16.059075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.058913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:16.059075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.058957 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:16.061552 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.061523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1558c0c0-4b52-4771-b01a-79c8786560c6-node-exporter-tls\") pod \"node-exporter-bhrsk\" (UID: \"1558c0c0-4b52-4771-b01a-79c8786560c6\") " pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:16.061712 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.061564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac42c982-7b05-4ba5-a82a-b1e9a1668eb1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ck5dw\" (UID: \"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:16.061712 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.061594 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf39828-cdf6-495e-96ee-b110e01dce91-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzjv7\" (UID: \"3cf39828-cdf6-495e-96ee-b110e01dce91\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:16.233466 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.233428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.319309 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.319275 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" Jul 22 01:46:16.331867 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.331831 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.339933 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.339902 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" Jul 22 01:46:16.356096 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.356062 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhrsk" Jul 22 01:46:16.386782 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.386742 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:46:16.407463 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.407428 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.458953 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.458884 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:46:16.459106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.459059 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.461750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.461728 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 01:46:16.461750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.461742 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8m9hd\"" Jul 22 01:46:16.461996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.461821 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 01:46:16.461996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.461859 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 01:46:16.462146 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462125 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 01:46:16.462248 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462224 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 01:46:16.462353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462331 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 01:46:16.462401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462362 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 01:46:16.462592 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462571 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 01:46:16.462681 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.462663 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 01:46:16.518901 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:16.518872 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1558c0c0_4b52_4771_b01a_79c8786560c6.slice/crio-62574daaa087b73ae5428df6958445a02bb73288595cf41ec57aa7394c36d8de WatchSource:0}: Error finding container 62574daaa087b73ae5428df6958445a02bb73288595cf41ec57aa7394c36d8de: Status 404 returned error can't find the container with id 62574daaa087b73ae5428df6958445a02bb73288595cf41ec57aa7394c36d8de Jul 22 01:46:16.563206 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563206 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563206 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563380 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563242 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563380 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563380 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563293 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqv7g\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563380 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563319 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563490 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563523 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563606 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.563817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.563676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.590268 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.590206 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.653422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.653390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw"] Jul 22 01:46:16.656342 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:16.656312 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac42c982_7b05_4ba5_a82a_b1e9a1668eb1.slice/crio-fa4264cdf0f9bb1cea7469a3715ddf155420154f21aceaec8bfccdae52eba638 WatchSource:0}: Error finding container fa4264cdf0f9bb1cea7469a3715ddf155420154f21aceaec8bfccdae52eba638: Status 404 returned error can't find the container with id fa4264cdf0f9bb1cea7469a3715ddf155420154f21aceaec8bfccdae52eba638 Jul 22 01:46:16.664997 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.664968 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665110 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665110 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665110 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665062 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665110 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665082 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665282 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665328 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.665479 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:16.665415 2585 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 22 01:46:16.665527 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:16.665496 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls podName:b01f9309-05ee-401c-866c-0d31e80279d3 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:17.165472746 +0000 UTC m=+55.066351236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3") : secret "alertmanager-main-tls" not found Jul 22 01:46:16.665585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.666172 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.666172 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665725 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.666172 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqv7g\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.666172 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.666172 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.665993 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.667902 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.667077 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.669176 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.669114 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.669464 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.669280 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.669464 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.669426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.670120 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.670010 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.670120 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.670089 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.670120 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.670094 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.670975 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.670955 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.671369 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.671327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.673291 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:16.673266 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf39828_cdf6_495e_96ee_b110e01dce91.slice/crio-a65521988f66c791129fc6b1e4f775d8e4f83e2e74a202df7ecb40ca9c8e6b0a WatchSource:0}: Error finding container a65521988f66c791129fc6b1e4f775d8e4f83e2e74a202df7ecb40ca9c8e6b0a: Status 404 returned error can't find the container with id a65521988f66c791129fc6b1e4f775d8e4f83e2e74a202df7ecb40ca9c8e6b0a Jul 22 01:46:16.675348 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.675324 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzjv7"] Jul 22 01:46:16.676417 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.676398 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqv7g\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:16.785892 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.785859 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:16.955268 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.955240 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhrsk" event={"ID":"1558c0c0-4b52-4771-b01a-79c8786560c6","Type":"ContainerStarted","Data":"62574daaa087b73ae5428df6958445a02bb73288595cf41ec57aa7394c36d8de"} Jul 22 01:46:16.957039 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.957006 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" event={"ID":"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1","Type":"ContainerStarted","Data":"e419b32249b45ec9be2f1f1a91b98d367b84bef44ffab8792e1022f7b7b43cb3"} Jul 22 01:46:16.957153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.957075 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" event={"ID":"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1","Type":"ContainerStarted","Data":"fa4264cdf0f9bb1cea7469a3715ddf155420154f21aceaec8bfccdae52eba638"} Jul 22 01:46:16.958141 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:16.958119 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" event={"ID":"3cf39828-cdf6-495e-96ee-b110e01dce91","Type":"ContainerStarted","Data":"a65521988f66c791129fc6b1e4f775d8e4f83e2e74a202df7ecb40ca9c8e6b0a"} Jul 22 01:46:17.170697 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.170606 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:17.173497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.173465 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:17.368451 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.368233 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:46:17.392027 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.391997 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:17.479845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.479809 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:17.588175 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.588131 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:17.596060 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.595955 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:46:17.598926 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:17.598893 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01f9309_05ee_401c_866c_0d31e80279d3.slice/crio-a8ceabaac121dc371c7d487e15905ece1b98bee7f270c93a63d5c5c7447d8dad WatchSource:0}: Error finding container a8ceabaac121dc371c7d487e15905ece1b98bee7f270c93a63d5c5c7447d8dad: Status 404 returned error can't find the container with id a8ceabaac121dc371c7d487e15905ece1b98bee7f270c93a63d5c5c7447d8dad Jul 22 01:46:17.688454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.688410 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:17.964860 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.964802 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" event={"ID":"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1","Type":"ContainerStarted","Data":"2cbe3523fe6df3dae2624ae7e92a3722557fe93efdf438bad75305780e7ff1de"} Jul 22 01:46:17.966236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.966201 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"a8ceabaac121dc371c7d487e15905ece1b98bee7f270c93a63d5c5c7447d8dad"} Jul 22 01:46:17.968090 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.968062 2585 generic.go:356] "Generic (PLEG): container finished" podID="1558c0c0-4b52-4771-b01a-79c8786560c6" containerID="73c3d8595fe5b9fe1ce3970c1a4d692e14a770c92f78ddf783e0acec15cfecce" exitCode=0 Jul 22 01:46:17.968240 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.968101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhrsk" event={"ID":"1558c0c0-4b52-4771-b01a-79c8786560c6","Type":"ContainerDied","Data":"73c3d8595fe5b9fe1ce3970c1a4d692e14a770c92f78ddf783e0acec15cfecce"} Jul 22 01:46:17.970385 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:17.970363 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-prvz2" event={"ID":"b9332bbf-18e3-408e-8051-b2cc2b31ac69","Type":"ContainerStarted","Data":"23677fd7a89ec56ea8a07b9f4b45caaa2fe05fd7926f9ed20fff533a13189276"} Jul 22 01:46:18.008983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.008934 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-prvz2" podStartSLOduration=2.110068681 podStartE2EDuration="19.008910926s" totalImagesPullingTime="16.898842245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:18.008495931 +0000 UTC m=+55.909374431" watchObservedRunningTime="2026-07-22 01:46:18.008910926 +0000 UTC m=+55.909789400" Jul 22 01:46:18.295456 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.295424 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74464c75bc-9z7lk"] Jul 22 01:46:18.299380 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.299277 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.302117 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.302038 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 01:46:18.302583 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.302562 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 01:46:18.302703 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.302677 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 01:46:18.302881 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.302842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9t35o3l3s8pe0\"" Jul 22 01:46:18.302988 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.302970 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 01:46:18.303107 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.303044 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 01:46:18.303487 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.303471 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rhpc8\"" Jul 22 01:46:18.311462 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.311440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74464c75bc-9z7lk"] Jul 22 01:46:18.357839 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.357711 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:18.381687 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.381655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.381844 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.381791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-grpc-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.381930 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45443b09-44f5-4998-b8dd-80265857ec20-metrics-client-ca\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.381963 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.381983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69w7\" (UniqueName: \"kubernetes.io/projected/45443b09-44f5-4998-b8dd-80265857ec20-kube-api-access-p69w7\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.382001 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.382109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.382249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.382147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.459868 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.459116 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:18.483427 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483303 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45443b09-44f5-4998-b8dd-80265857ec20-metrics-client-ca\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483427 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69w7\" (UniqueName: \"kubernetes.io/projected/45443b09-44f5-4998-b8dd-80265857ec20-kube-api-access-p69w7\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483722 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483831 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483831 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483768 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483831 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483811 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.483983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.483872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-grpc-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.484390 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.484258 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45443b09-44f5-4998-b8dd-80265857ec20-metrics-client-ca\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.487422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.487386 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.487422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.487389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.487629 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.487449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.487629 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.487501 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.488435 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.488390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-grpc-tls\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.488853 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.488830 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45443b09-44f5-4998-b8dd-80265857ec20-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.494022 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.493994 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69w7\" (UniqueName: \"kubernetes.io/projected/45443b09-44f5-4998-b8dd-80265857ec20-kube-api-access-p69w7\") pod \"thanos-querier-74464c75bc-9z7lk\" (UID: \"45443b09-44f5-4998-b8dd-80265857ec20\") " pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.625243 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.625154 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:18.647878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.647842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:18.791021 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.790986 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74464c75bc-9z7lk"] Jul 22 01:46:18.794766 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:18.794734 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45443b09_44f5_4998_b8dd_80265857ec20.slice/crio-bf92171b69110c10f74318d6489c51dc22cbb424ca95d0fcb86b45acd98e60af WatchSource:0}: Error finding container bf92171b69110c10f74318d6489c51dc22cbb424ca95d0fcb86b45acd98e60af: Status 404 returned error can't find the container with id bf92171b69110c10f74318d6489c51dc22cbb424ca95d0fcb86b45acd98e60af Jul 22 01:46:18.920452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.920427 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gnwr2" Jul 22 01:46:18.976463 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.976430 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhrsk" event={"ID":"1558c0c0-4b52-4771-b01a-79c8786560c6","Type":"ContainerStarted","Data":"dab8c4ea61ebb28f3e794257da89f634f33e15e2ca2dadb8f371097341391f95"} Jul 22 01:46:18.976921 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.976488 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhrsk" event={"ID":"1558c0c0-4b52-4771-b01a-79c8786560c6","Type":"ContainerStarted","Data":"f47459dadf6a81d98008b6afdf8450165f3bdd82a993ecfb83df62e77b710729"} Jul 22 01:46:18.977882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.977853 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"bf92171b69110c10f74318d6489c51dc22cbb424ca95d0fcb86b45acd98e60af"} Jul 22 01:46:18.980357 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.980318 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" event={"ID":"ac42c982-7b05-4ba5-a82a-b1e9a1668eb1","Type":"ContainerStarted","Data":"e4af035844d5519128b26eba819ba47ab905f17f6fca84b72c4ced866855c669"} Jul 22 01:46:18.982522 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.982499 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" event={"ID":"3cf39828-cdf6-495e-96ee-b110e01dce91","Type":"ContainerStarted","Data":"eec8fa5a51599c9141f05ce3183b4d33af3a460a14989b77ed079cd7409055a6"} Jul 22 01:46:18.982608 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.982559 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" event={"ID":"3cf39828-cdf6-495e-96ee-b110e01dce91","Type":"ContainerStarted","Data":"f8194f9c0b215ad42a3665202ae8d28baf17f1d63d5190fbf1402666497113d2"} Jul 22 01:46:18.982608 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.982590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" event={"ID":"3cf39828-cdf6-495e-96ee-b110e01dce91","Type":"ContainerStarted","Data":"092b7c99da80edc79ef934b0fd47d4b3af6c2f89470421b3598262741a1aed1d"} Jul 22 01:46:18.995736 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:18.995671 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bhrsk" podStartSLOduration=2.9682731479999998 podStartE2EDuration="3.995626952s" totalImagesPullingTime="939.206328ms" totalInitContainerRuntime="88.147476ms" isStatefulPod=true podCreationTimestamp="2026-07-22 01:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:18.994696793 +0000 UTC m=+56.895575332" watchObservedRunningTime="2026-07-22 01:46:18.995626952 +0000 UTC m=+56.896505450" Jul 22 01:46:19.014818 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.014755 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ck5dw" podStartSLOduration=2.737186008 podStartE2EDuration="4.01473129s" totalImagesPullingTime="1.277545282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:19.013006853 +0000 UTC m=+56.913885364" watchObservedRunningTime="2026-07-22 01:46:19.01473129 +0000 UTC m=+56.915609789" Jul 22 01:46:19.032085 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.032029 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzjv7" podStartSLOduration=2.427247785 podStartE2EDuration="4.032013686s" totalImagesPullingTime="1.604765901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:19.030850003 +0000 UTC m=+56.931728513" watchObservedRunningTime="2026-07-22 01:46:19.032013686 +0000 UTC m=+56.932892183" Jul 22 01:46:19.206108 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.206020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:19.312879 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.312848 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:19.665714 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.665676 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c9bf68595-vdlpl"] Jul 22 01:46:19.668163 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.668146 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.670770 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.670741 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 01:46:19.672144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.671982 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cscqc\"" Jul 22 01:46:19.672144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.671983 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 01:46:19.672144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.672039 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 01:46:19.672144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.672098 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5e6jvjetgc7t5\"" Jul 22 01:46:19.672144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.672122 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 01:46:19.679072 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.679040 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c9bf68595-vdlpl"] Jul 22 01:46:19.694761 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-metrics-server-audit-profiles\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.694761 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8278cb5b-c1e1-4457-abff-e4ee88956ad2-audit-log\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.694761 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-client-certs\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.695097 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9hzg\" (UniqueName: \"kubernetes.io/projected/8278cb5b-c1e1-4457-abff-e4ee88956ad2-kube-api-access-k9hzg\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.695097 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694921 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.695097 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.694980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-tls\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.695097 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.695030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-client-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.707934 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.707887 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:19.708118 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.707962 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:19.713901 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.713874 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:19.795711 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.795671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-client-certs\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.795909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.795752 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9hzg\" (UniqueName: \"kubernetes.io/projected/8278cb5b-c1e1-4457-abff-e4ee88956ad2-kube-api-access-k9hzg\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.795909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.795784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.795909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.795843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-tls\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.796325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.796286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-client-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.796443 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.796424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-metrics-server-audit-profiles\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.796546 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.796521 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.796619 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.796550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8278cb5b-c1e1-4457-abff-e4ee88956ad2-audit-log\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.796959 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.796913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8278cb5b-c1e1-4457-abff-e4ee88956ad2-audit-log\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.797531 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.797503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8278cb5b-c1e1-4457-abff-e4ee88956ad2-metrics-server-audit-profiles\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.798817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.798791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-client-certs\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.798906 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.798861 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-secret-metrics-server-tls\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.799908 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.799884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8278cb5b-c1e1-4457-abff-e4ee88956ad2-client-ca-bundle\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.805078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.805015 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9hzg\" (UniqueName: \"kubernetes.io/projected/8278cb5b-c1e1-4457-abff-e4ee88956ad2-kube-api-access-k9hzg\") pod \"metrics-server-7c9bf68595-vdlpl\" (UID: \"8278cb5b-c1e1-4457-abff-e4ee88956ad2\") " pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.977740 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.977613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:19.989304 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.989122 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd" exitCode=0 Jul 22 01:46:19.989304 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.989192 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd"} Jul 22 01:46:19.995091 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:19.995056 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:46:20.066209 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.066175 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:20.125301 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.125240 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:20.127183 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.127132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:20.128585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.128564 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.136345 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.136003 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l"] Jul 22 01:46:20.138995 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.138832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.139361 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.139338 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:20.141805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.141650 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 01:46:20.142375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.142354 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ch7zl\"" Jul 22 01:46:20.149367 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.148790 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l"] Jul 22 01:46:20.157161 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.157090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c9bf68595-vdlpl"] Jul 22 01:46:20.157414 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:20.157383 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8278cb5b_c1e1_4457_abff_e4ee88956ad2.slice/crio-c09ab443f19944f648b318c46fd54978b1197b840f469f536915db940fff51a6 WatchSource:0}: Error finding container c09ab443f19944f648b318c46fd54978b1197b840f469f536915db940fff51a6: Status 404 returned error can't find the container with id c09ab443f19944f648b318c46fd54978b1197b840f469f536915db940fff51a6 Jul 22 01:46:20.200845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.200807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.200845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.200849 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.201086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.200920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.201086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.200967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmj2k\" (UniqueName: \"kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.201086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.200998 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4z6\" (UniqueName: \"kubernetes.io/projected/f3418968-fff5-43e9-8c51-47e2ad6dbb91-kube-api-access-5b4z6\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.201086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.201033 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.201086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.201081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.201317 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.201112 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.201317 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.201165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.269895 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.269818 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:20.302002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.301965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302034 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmj2k\" (UniqueName: \"kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4z6\" (UniqueName: \"kubernetes.io/projected/f3418968-fff5-43e9-8c51-47e2ad6dbb91-kube-api-access-5b4z6\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:20.302119 2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302132 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:20.302195 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert podName:f3418968-fff5-43e9-8c51-47e2ad6dbb91 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:20.802174102 +0000 UTC m=+58.703052579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-lgj5l" (UID: "f3418968-fff5-43e9-8c51-47e2ad6dbb91") : secret "monitoring-plugin-cert" not found Jul 22 01:46:20.302375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.303059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.302982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.303296 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.303239 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.303296 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.303248 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.303481 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.303460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.305211 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.305178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.305305 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.305291 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.311153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.311128 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4z6\" (UniqueName: \"kubernetes.io/projected/f3418968-fff5-43e9-8c51-47e2ad6dbb91-kube-api-access-5b4z6\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.311263 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.311126 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmj2k\" (UniqueName: \"kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k\") pod \"console-58b56b47db-fjfzp\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.442791 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.442737 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:20.510014 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.509966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:20.572177 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.572136 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86769bcb8d-sm4q8"] Jul 22 01:46:20.581951 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.578748 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.584670 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.584605 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 01:46:20.585220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.585035 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 01:46:20.585220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.585096 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 01:46:20.586542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.586010 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 01:46:20.586542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.586305 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 01:46:20.586754 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.586730 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kscfb\"" Jul 22 01:46:20.587706 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.587684 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86769bcb8d-sm4q8"] Jul 22 01:46:20.591317 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.591286 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 01:46:20.607655 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcpf\" (UniqueName: \"kubernetes.io/projected/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-kube-api-access-6fcpf\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.607655 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607593 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-metrics-client-ca\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.607880 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.607880 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.607880 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-federate-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.607880 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.608078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607915 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-serving-certs-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.608078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.607980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.617350 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.617318 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:20.709461 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcpf\" (UniqueName: \"kubernetes.io/projected/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-kube-api-access-6fcpf\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709477 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-metrics-client-ca\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-federate-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709893 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.709893 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.709878 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-serving-certs-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.710086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.710047 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.711565 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.710344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-metrics-client-ca\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.711565 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.711020 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.711565 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.711507 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-serving-certs-ca-bundle\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.712442 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.712300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.712566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.712549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-federate-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.712845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.712825 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-telemeter-client-tls\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.713945 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.713925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.717938 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.717911 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcpf\" (UniqueName: \"kubernetes.io/projected/88428074-9b3c-4d7d-9c78-7d1a58e34fd7-kube-api-access-6fcpf\") pod \"telemeter-client-86769bcb8d-sm4q8\" (UID: \"88428074-9b3c-4d7d-9c78-7d1a58e34fd7\") " pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.761704 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.761589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:20.811444 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.811349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.814012 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.813982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3418968-fff5-43e9-8c51-47e2ad6dbb91-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-lgj5l\" (UID: \"f3418968-fff5-43e9-8c51-47e2ad6dbb91\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:20.894959 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.894924 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" Jul 22 01:46:20.995454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:20.995412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" event={"ID":"8278cb5b-c1e1-4457-abff-e4ee88956ad2","Type":"ContainerStarted","Data":"c09ab443f19944f648b318c46fd54978b1197b840f469f536915db940fff51a6"} Jul 22 01:46:21.054215 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.054177 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:21.145902 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:21.145820 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa9edf_087d_40dc_b794_6e7d9280165d.slice/crio-12d72b9b8c704826e1a0fece2371c0bd7dee4beb562f5b985a2e5fca11da42f2 WatchSource:0}: Error finding container 12d72b9b8c704826e1a0fece2371c0bd7dee4beb562f5b985a2e5fca11da42f2: Status 404 returned error can't find the container with id 12d72b9b8c704826e1a0fece2371c0bd7dee4beb562f5b985a2e5fca11da42f2 Jul 22 01:46:21.706254 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.705542 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:21.709544 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.707993 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:46:21.716669 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.716628 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.720982 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.721210 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2irkv8o4kpl\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.721256 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.721008 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.721995 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.722367 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 01:46:21.722763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.722578 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 01:46:21.723196 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.722783 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.726254 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.726541 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.726600 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.726762 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.726976 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 01:46:21.727661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.727500 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z4wf8\"" Jul 22 01:46:21.731841 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.731421 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 01:46:21.743817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.743726 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:46:21.765358 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.764620 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823822 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823948 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.823971 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824233 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824353 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6kd\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.825996 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.824429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.851759 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.851610 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5w5c" Jul 22 01:46:21.867482 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.863485 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86769bcb8d-sm4q8"] Jul 22 01:46:21.873210 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:21.870952 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88428074_9b3c_4d7d_9c78_7d1a58e34fd7.slice/crio-68542c25aba5eb1e11ed494a9d65e9bf070e2472a2c2088b8421316a90081456 WatchSource:0}: Error finding container 68542c25aba5eb1e11ed494a9d65e9bf070e2472a2c2088b8421316a90081456: Status 404 returned error can't find the container with id 68542c25aba5eb1e11ed494a9d65e9bf070e2472a2c2088b8421316a90081456 Jul 22 01:46:21.897660 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.897544 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l"] Jul 22 01:46:21.901988 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:21.901954 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3418968_fff5_43e9_8c51_47e2ad6dbb91.slice/crio-e2fcaee2f85748fe632cbcd9dba062e9783911e791cb1ae66d35dd65b99dd8f2 WatchSource:0}: Error finding container e2fcaee2f85748fe632cbcd9dba062e9783911e791cb1ae66d35dd65b99dd8f2: Status 404 returned error can't find the container with id e2fcaee2f85748fe632cbcd9dba062e9783911e791cb1ae66d35dd65b99dd8f2 Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.926854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.926897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.926925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.926951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.926975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927418 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6kd\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.927851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927781 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.927852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.930270 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.932519 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.932786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.933497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.933044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.937533 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.936361 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.939419 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.938226 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.939419 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.938940 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.941113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.939713 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.941113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.940270 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.941113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.940702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.941972 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.941795 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.943045 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.942996 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.944563 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.944109 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.946059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.945265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.946059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.945269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.946059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.945617 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.946059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.945691 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.946059 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.946024 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.949818 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.949781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6kd\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd\") pod \"prometheus-k8s-0\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:21.992986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:21.988668 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:22.005758 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.004923 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b56b47db-fjfzp" event={"ID":"deaa9edf-087d-40dc-b794-6e7d9280165d","Type":"ContainerStarted","Data":"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f"} Jul 22 01:46:22.005758 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.004986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b56b47db-fjfzp" event={"ID":"deaa9edf-087d-40dc-b794-6e7d9280165d","Type":"ContainerStarted","Data":"12d72b9b8c704826e1a0fece2371c0bd7dee4beb562f5b985a2e5fca11da42f2"} Jul 22 01:46:22.006785 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.006751 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" event={"ID":"88428074-9b3c-4d7d-9c78-7d1a58e34fd7","Type":"ContainerStarted","Data":"68542c25aba5eb1e11ed494a9d65e9bf070e2472a2c2088b8421316a90081456"} Jul 22 01:46:22.011603 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.011396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825"} Jul 22 01:46:22.011603 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.011533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f"} Jul 22 01:46:22.015442 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.015408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"08fbf036797163718b6ab1d1a211a5308c46b5f577adeb2788bb78c2b9096aac"} Jul 22 01:46:22.015555 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.015485 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"554163aa72e2914d7e45687a592bc450baf0c599c30641c54755c74e34a9eed3"} Jul 22 01:46:22.015555 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.015526 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"d3ef0f96472f0df4fb8146f6bbdc0cc385484352a406d72ce5a2b289d813ef31"} Jul 22 01:46:22.016802 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.016743 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" event={"ID":"f3418968-fff5-43e9-8c51-47e2ad6dbb91","Type":"ContainerStarted","Data":"e2fcaee2f85748fe632cbcd9dba062e9783911e791cb1ae66d35dd65b99dd8f2"} Jul 22 01:46:22.038670 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.037595 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b56b47db-fjfzp" podStartSLOduration=2.037574756 podStartE2EDuration="2.037574756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:22.03612593 +0000 UTC m=+59.937004428" watchObservedRunningTime="2026-07-22 01:46:22.037574756 +0000 UTC m=+59.938453253" Jul 22 01:46:22.091344 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.091146 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:22.093129 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.092910 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:22.142401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.142269 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:22.189445 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.189385 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:22.689567 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.689544 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:46:22.691393 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:22.691365 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded415836_0d1b_4276_b743_d9e4cd714eb7.slice/crio-85794917120bf7bd30a783241c08f20ec65773717deae266c1ad6b479cbb900b WatchSource:0}: Error finding container 85794917120bf7bd30a783241c08f20ec65773717deae266c1ad6b479cbb900b: Status 404 returned error can't find the container with id 85794917120bf7bd30a783241c08f20ec65773717deae266c1ad6b479cbb900b Jul 22 01:46:22.794069 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.794034 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:22.893982 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:22.893957 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:23.024310 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.024221 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" event={"ID":"8278cb5b-c1e1-4457-abff-e4ee88956ad2","Type":"ContainerStarted","Data":"deca0dbb65b26b5c1d7df10afbb0ef2f3bd24b5b53e2df4530218bfd8fd93716"} Jul 22 01:46:23.042333 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.042281 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e"} Jul 22 01:46:23.042713 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.042690 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072"} Jul 22 01:46:23.042995 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.042978 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad"} Jul 22 01:46:23.048266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.047493 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43" exitCode=0 Jul 22 01:46:23.048266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.048122 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43"} Jul 22 01:46:23.048266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.048180 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"85794917120bf7bd30a783241c08f20ec65773717deae266c1ad6b479cbb900b"} Jul 22 01:46:23.081739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:23.080554 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" podStartSLOduration=1.356729382 podStartE2EDuration="4.080532358s" totalImagesPullingTime="2.723802976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:23.045767795 +0000 UTC m=+60.946646289" watchObservedRunningTime="2026-07-22 01:46:23.080532358 +0000 UTC m=+60.981410858" Jul 22 01:46:24.121369 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:24.121328 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:24.222213 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:24.222177 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:25.062321 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.062284 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerStarted","Data":"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8"} Jul 22 01:46:25.066237 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.066204 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"e71220ad6f9df8e0c05abf04620f760e0c4c4d7ba648fe8a802485ea915ab426"} Jul 22 01:46:25.066401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.066279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"38984fb41a21bf57ebde3e80f46be6c93eb87639ec6ae6d4955c9fff57d52862"} Jul 22 01:46:25.066401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.066317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" event={"ID":"45443b09-44f5-4998-b8dd-80265857ec20","Type":"ContainerStarted","Data":"6c1cda974e92492f134fc1593268bdc382d6fc3a6f8bf8ec0fe9b81299f594f0"} Jul 22 01:46:25.066527 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.066459 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:25.068359 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.068307 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" event={"ID":"f3418968-fff5-43e9-8c51-47e2ad6dbb91","Type":"ContainerStarted","Data":"44e73795906d7eee8601ace4c28af8ed1835538c4a9595e7358e1e0836eda467"} Jul 22 01:46:25.068501 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.068480 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:25.070767 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.070737 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" event={"ID":"88428074-9b3c-4d7d-9c78-7d1a58e34fd7","Type":"ContainerStarted","Data":"f78701a9b66ee4fc16db762a19328d8e68f53797379ea093ba3fed76aa0adc84"} Jul 22 01:46:25.070883 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.070792 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" event={"ID":"88428074-9b3c-4d7d-9c78-7d1a58e34fd7","Type":"ContainerStarted","Data":"9e7d0e9e2cd8320ca55f498d4e5aaaa61ae9a3ce1e2e453356b8b0cc616689be"} Jul 22 01:46:25.070883 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.070821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" event={"ID":"88428074-9b3c-4d7d-9c78-7d1a58e34fd7","Type":"ContainerStarted","Data":"68fd06ba9c93586d64444eee423a89e9dd152ad14cced80a04562ac4dfa88517"} Jul 22 01:46:25.074336 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.074316 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" Jul 22 01:46:25.092420 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.092360 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.827911832 podStartE2EDuration="9.092339804s" totalImagesPullingTime="5.247460107s" totalInitContainerRuntime="16.967865ms" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:25.089286369 +0000 UTC m=+62.990164866" watchObservedRunningTime="2026-07-22 01:46:25.092339804 +0000 UTC m=+62.993218302" Jul 22 01:46:25.112906 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.112797 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86769bcb8d-sm4q8" podStartSLOduration=2.375724505 podStartE2EDuration="5.112774148s" totalImagesPullingTime="2.737049643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:25.110919652 +0000 UTC m=+63.011798151" watchObservedRunningTime="2026-07-22 01:46:25.112774148 +0000 UTC m=+63.013652626" Jul 22 01:46:25.126855 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.126798 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-lgj5l" podStartSLOduration=2.418463245 podStartE2EDuration="5.126779258s" totalImagesPullingTime="2.708316013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:25.125807556 +0000 UTC m=+63.026686097" watchObservedRunningTime="2026-07-22 01:46:25.126779258 +0000 UTC m=+63.027657759" Jul 22 01:46:25.150213 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.150159 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" podStartSLOduration=1.675573584 podStartE2EDuration="7.1501355s" totalImagesPullingTime="5.474561916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:25.149687254 +0000 UTC m=+63.050565753" watchObservedRunningTime="2026-07-22 01:46:25.1501355 +0000 UTC m=+63.051013998" Jul 22 01:46:25.622521 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.622480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:25.857458 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.857417 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:25.894447 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.894002 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:46:25.898041 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.898011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.905970 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.903210 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:46:25.977366 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.977622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.977801 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977770 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.977912 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.977912 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977875 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.978030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977919 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:25.978030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:25.977952 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghhw\" (UniqueName: \"kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghhw\" (UniqueName: \"kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.078876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.079373 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.079564 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.079903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.081434 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.080147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.082113 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.081785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.082354 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.082336 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.084247 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.084226 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74464c75bc-9z7lk" Jul 22 01:46:26.089144 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.089124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghhw\" (UniqueName: \"kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw\") pod \"console-7966975879-vfkf6\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:26.212612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:26.212573 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:27.103886 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:27.103855 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:46:28.035209 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.035176 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:28.086814 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.086770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274"} Jul 22 01:46:28.086983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.086856 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d"} Jul 22 01:46:28.086983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.086898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1"} Jul 22 01:46:28.086983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.086939 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab"} Jul 22 01:46:28.086983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.086974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027"} Jul 22 01:46:28.087188 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.087010 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerStarted","Data":"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5"} Jul 22 01:46:28.088390 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.088368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966975879-vfkf6" event={"ID":"09d82c40-7f76-4732-ad94-70c74939c7e2","Type":"ContainerStarted","Data":"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988"} Jul 22 01:46:28.088497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.088405 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966975879-vfkf6" event={"ID":"09d82c40-7f76-4732-ad94-70c74939c7e2","Type":"ContainerStarted","Data":"7a319bb1b598e1e61302265709de31350f9ca77496a0decd17be06597e1c32f5"} Jul 22 01:46:28.119800 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.119752 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.15302821 podStartE2EDuration="7.119731264s" totalImagesPullingTime="3.948723839s" totalInitContainerRuntime="17.979215ms" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:28.116955613 +0000 UTC m=+66.017834107" watchObservedRunningTime="2026-07-22 01:46:28.119731264 +0000 UTC m=+66.020609761" Jul 22 01:46:28.134459 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.134400 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7966975879-vfkf6" podStartSLOduration=3.134380029 podStartE2EDuration="3.134380029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:28.133782589 +0000 UTC m=+66.034661087" watchObservedRunningTime="2026-07-22 01:46:28.134380029 +0000 UTC m=+66.035258526" Jul 22 01:46:28.209115 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.209043 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:46:28.211995 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.211969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:46:28.221816 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.221791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e511869-a8c9-4c7f-8926-cff6e15d2849-original-pull-secret\") pod \"global-pull-secret-syncer-9xd9q\" (UID: \"9e511869-a8c9-4c7f-8926-cff6e15d2849\") " pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:46:28.299879 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.299789 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xd9q" Jul 22 01:46:28.309965 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.309933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:46:28.312791 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.312753 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:46:28.323460 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.323430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eabe7ab-9a59-4616-8637-b01ac6367c63-metrics-certs\") pod \"network-metrics-daemon-sgw7b\" (UID: \"8eabe7ab-9a59-4616-8637-b01ac6367c63\") " pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:46:28.411428 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.411392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:46:28.414353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.414329 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:46:28.424334 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.424307 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:46:28.431647 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.431614 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xd9q"] Jul 22 01:46:28.434122 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:28.434088 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e511869_a8c9_4c7f_8926_cff6e15d2849.slice/crio-5d747fc3e9415eb244d8bb8c7918953a9381406e195385052505afc77f8804ea WatchSource:0}: Error finding container 5d747fc3e9415eb244d8bb8c7918953a9381406e195385052505afc77f8804ea: Status 404 returned error can't find the container with id 5d747fc3e9415eb244d8bb8c7918953a9381406e195385052505afc77f8804ea Jul 22 01:46:28.435391 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.435371 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzjv\" (UniqueName: \"kubernetes.io/projected/724b8752-2c6f-45d9-a545-7310096a80da-kube-api-access-6mzjv\") pod \"network-check-target-6bqbj\" (UID: \"724b8752-2c6f-45d9-a545-7310096a80da\") " pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:46:28.486422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.486391 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wgwz\"" Jul 22 01:46:28.494745 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.494717 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:46:28.595325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.595295 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cgfms\"" Jul 22 01:46:28.603759 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.603692 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sgw7b" Jul 22 01:46:28.620835 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.620813 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bqbj"] Jul 22 01:46:28.622514 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:28.622493 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724b8752_2c6f_45d9_a545_7310096a80da.slice/crio-87be33b90d60b33bd835d0f72f495fd2f77878b9c0c99bf68a28bb92e8fec41a WatchSource:0}: Error finding container 87be33b90d60b33bd835d0f72f495fd2f77878b9c0c99bf68a28bb92e8fec41a: Status 404 returned error can't find the container with id 87be33b90d60b33bd835d0f72f495fd2f77878b9c0c99bf68a28bb92e8fec41a Jul 22 01:46:28.737396 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:28.737372 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sgw7b"] Jul 22 01:46:28.739661 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:46:28.739621 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eabe7ab_9a59_4616_8637_b01ac6367c63.slice/crio-81380ea4c37d2965da80d408f87a2151276fb80d9e862268e1feab569b3399d2 WatchSource:0}: Error finding container 81380ea4c37d2965da80d408f87a2151276fb80d9e862268e1feab569b3399d2: Status 404 returned error can't find the container with id 81380ea4c37d2965da80d408f87a2151276fb80d9e862268e1feab569b3399d2 Jul 22 01:46:29.096204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:29.096165 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgw7b" event={"ID":"8eabe7ab-9a59-4616-8637-b01ac6367c63","Type":"ContainerStarted","Data":"81380ea4c37d2965da80d408f87a2151276fb80d9e862268e1feab569b3399d2"} Jul 22 01:46:29.097651 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:29.097594 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bqbj" event={"ID":"724b8752-2c6f-45d9-a545-7310096a80da","Type":"ContainerStarted","Data":"87be33b90d60b33bd835d0f72f495fd2f77878b9c0c99bf68a28bb92e8fec41a"} Jul 22 01:46:29.098981 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:29.098953 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xd9q" event={"ID":"9e511869-a8c9-4c7f-8926-cff6e15d2849","Type":"ContainerStarted","Data":"5d747fc3e9415eb244d8bb8c7918953a9381406e195385052505afc77f8804ea"} Jul 22 01:46:30.443713 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:30.443683 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:31.111853 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:31.111815 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgw7b" event={"ID":"8eabe7ab-9a59-4616-8637-b01ac6367c63","Type":"ContainerStarted","Data":"dbbe1ed9876004cb0a33beee1d71d029fc1890471d9962db35ea52d055ea5cc5"} Jul 22 01:46:31.112030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:31.111876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sgw7b" event={"ID":"8eabe7ab-9a59-4616-8637-b01ac6367c63","Type":"ContainerStarted","Data":"30a79e60b9311a25ba69b1d02372f77795d3203e3e68f27569b62cbd1b6fe586"} Jul 22 01:46:31.130340 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:31.130285 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sgw7b" podStartSLOduration=67.75838413 podStartE2EDuration="1m9.130266149s" totalImagesPullingTime="1.371882019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:31.127308512 +0000 UTC m=+69.028187010" watchObservedRunningTime="2026-07-22 01:46:31.130266149 +0000 UTC m=+69.031144646" Jul 22 01:46:31.217063 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:31.217027 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:31.280239 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:31.280200 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:32.093340 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:32.093305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:34.128575 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:34.128532 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bqbj" event={"ID":"724b8752-2c6f-45d9-a545-7310096a80da","Type":"ContainerStarted","Data":"5568268eebca8eb53b978fa5a6b1fb9860473b7a78d7a28fb791c5586e860bd4"} Jul 22 01:46:34.129558 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:34.129494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:46:34.131483 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:34.131447 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xd9q" event={"ID":"9e511869-a8c9-4c7f-8926-cff6e15d2849","Type":"ContainerStarted","Data":"1e542ecae8baa1c777544e27bc30d06750e44cbe8d1f422c9c53c93b3d69c410"} Jul 22 01:46:34.145793 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:34.145739 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6bqbj" podStartSLOduration=67.231388206 podStartE2EDuration="1m12.145718817s" totalImagesPullingTime="4.914330611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:34.144053691 +0000 UTC m=+72.044932190" watchObservedRunningTime="2026-07-22 01:46:34.145718817 +0000 UTC m=+72.046597313" Jul 22 01:46:34.159985 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:34.159938 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9xd9q" podStartSLOduration=67.045752581 podStartE2EDuration="1m12.159922357s" totalImagesPullingTime="5.114169776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:34.158107739 +0000 UTC m=+72.058986239" watchObservedRunningTime="2026-07-22 01:46:34.159922357 +0000 UTC m=+72.060800874" Jul 22 01:46:36.213065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:36.213026 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:36.213454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:36.213104 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:36.218452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:36.218428 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:37.153333 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.153302 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:46:37.200991 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.200959 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:46:37.306417 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.306384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:37.403034 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.403000 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:37.796382 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.796350 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:37.897060 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:37.897029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:39.978533 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:39.978494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:39.978958 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:39.978575 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:45.096813 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.096676 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f4cc4b4b5-n6ntm" podUID="b45141ee-5c79-418c-b252-b6ca59be6cb3" containerName="console" containerID="cri-o://3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352" gracePeriod=15 Jul 22 01:46:45.893696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.893660 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:45.982325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982331 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982573 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982389 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982573 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982503 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gwlm\" (UniqueName: \"kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982573 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982565 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982605 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert\") pod \"b45141ee-5c79-418c-b252-b6ca59be6cb3\" (UID: \"b45141ee-5c79-418c-b252-b6ca59be6cb3\") " Jul 22 01:46:45.982876 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982853 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca" (OuterVolumeSpecName: "service-ca") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:45.982942 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982859 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:45.982942 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982922 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config" (OuterVolumeSpecName: "console-config") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:45.983052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.982985 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:45.983052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.983033 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:45.985044 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.985019 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:46:45.985167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.985049 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm" (OuterVolumeSpecName: "kube-api-access-5gwlm") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "kube-api-access-5gwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:46:45.985167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:45.985080 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b45141ee-5c79-418c-b252-b6ca59be6cb3" (UID: "b45141ee-5c79-418c-b252-b6ca59be6cb3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:46:46.084389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.084309 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:46.084389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.084334 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gwlm\" (UniqueName: \"kubernetes.io/projected/b45141ee-5c79-418c-b252-b6ca59be6cb3-kube-api-access-5gwlm\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:46.084389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.084344 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:46.084389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.084354 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b45141ee-5c79-418c-b252-b6ca59be6cb3-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:46.187668 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.187617 2585 generic.go:356] "Generic (PLEG): container finished" podID="b45141ee-5c79-418c-b252-b6ca59be6cb3" containerID="3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352" exitCode=0 Jul 22 01:46:46.187668 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.187667 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4cc4b4b5-n6ntm" Jul 22 01:46:46.188147 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.187696 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4cc4b4b5-n6ntm" event={"ID":"b45141ee-5c79-418c-b252-b6ca59be6cb3","Type":"ContainerDied","Data":"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352"} Jul 22 01:46:46.188147 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.187768 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4cc4b4b5-n6ntm" event={"ID":"b45141ee-5c79-418c-b252-b6ca59be6cb3","Type":"ContainerDied","Data":"3eba6da27d02eeb0671a4b89dbfc8d4480d435a69e2a6e49d4c5a285b6906df0"} Jul 22 01:46:46.188147 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.187788 2585 scope.go:117] "RemoveContainer" containerID="3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352" Jul 22 01:46:46.197984 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.197966 2585 scope.go:117] "RemoveContainer" containerID="3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352" Jul 22 01:46:46.198319 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:46.198294 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352\": container with ID starting with 3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352 not found: ID does not exist" containerID="3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352" Jul 22 01:46:46.198370 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.198328 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352"} err="failed to get container status \"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352\": rpc error: code = NotFound desc = could not find container \"3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352\": container with ID starting with 3cbd1173864a32c445dea607b2cd9a9905e02a9e801c83cfc32c63109111a352 not found: ID does not exist" Jul 22 01:46:46.210564 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.210530 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:46.215236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.215208 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f4cc4b4b5-n6ntm"] Jul 22 01:46:46.677471 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:46.677442 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45141ee-5c79-418c-b252-b6ca59be6cb3" path="/var/lib/kubelet/pods/b45141ee-5c79-418c-b252-b6ca59be6cb3/volumes" Jul 22 01:46:47.199777 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:47.199746 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:47.262474 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:47.262448 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:48.213700 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:48.213629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:46:50.885858 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:50.885798 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58b56b47db-fjfzp" podUID="deaa9edf-087d-40dc-b794-6e7d9280165d" containerName="console" containerID="cri-o://2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f" gracePeriod=15 Jul 22 01:46:51.199497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.199463 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:51.213464 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.213435 2585 generic.go:356] "Generic (PLEG): container finished" podID="deaa9edf-087d-40dc-b794-6e7d9280165d" containerID="2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f" exitCode=0 Jul 22 01:46:51.213624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.213542 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b56b47db-fjfzp" event={"ID":"deaa9edf-087d-40dc-b794-6e7d9280165d","Type":"ContainerDied","Data":"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f"} Jul 22 01:46:51.213624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.213564 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b56b47db-fjfzp" Jul 22 01:46:51.213624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.213584 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b56b47db-fjfzp" event={"ID":"deaa9edf-087d-40dc-b794-6e7d9280165d","Type":"ContainerDied","Data":"12d72b9b8c704826e1a0fece2371c0bd7dee4beb562f5b985a2e5fca11da42f2"} Jul 22 01:46:51.213624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.213606 2585 scope.go:117] "RemoveContainer" containerID="2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f" Jul 22 01:46:51.223528 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.223503 2585 scope.go:117] "RemoveContainer" containerID="2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f" Jul 22 01:46:51.223863 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:46:51.223843 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f\": container with ID starting with 2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f not found: ID does not exist" containerID="2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f" Jul 22 01:46:51.223947 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.223872 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f"} err="failed to get container status \"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f\": rpc error: code = NotFound desc = could not find container \"2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f\": container with ID starting with 2b7508aac7e740e7b3d2bda8512ae18bc3bb20369b531766c5931a4874f8946f not found: ID does not exist" Jul 22 01:46:51.228161 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228140 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228243 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228187 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228243 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228224 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228245 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228265 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228291 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228316 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmj2k\" (UniqueName: \"kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k\") pod \"deaa9edf-087d-40dc-b794-6e7d9280165d\" (UID: \"deaa9edf-087d-40dc-b794-6e7d9280165d\") " Jul 22 01:46:51.228840 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228796 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca" (OuterVolumeSpecName: "service-ca") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:51.228840 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228807 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:51.228840 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228812 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config" (OuterVolumeSpecName: "console-config") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:51.229004 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.228862 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:46:51.230914 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.230889 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:46:51.231023 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.230893 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k" (OuterVolumeSpecName: "kube-api-access-jmj2k") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "kube-api-access-jmj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:46:51.231023 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.230968 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "deaa9edf-087d-40dc-b794-6e7d9280165d" (UID: "deaa9edf-087d-40dc-b794-6e7d9280165d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:46:51.329502 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329462 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329501 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329535 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329544 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329553 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmj2k\" (UniqueName: \"kubernetes.io/projected/deaa9edf-087d-40dc-b794-6e7d9280165d-kube-api-access-jmj2k\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329562 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa9edf-087d-40dc-b794-6e7d9280165d-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.329673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.329572 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa9edf-087d-40dc-b794-6e7d9280165d-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:46:51.542472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.542442 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:51.546823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:51.546796 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58b56b47db-fjfzp"] Jul 22 01:46:52.676426 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:52.676388 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deaa9edf-087d-40dc-b794-6e7d9280165d" path="/var/lib/kubelet/pods/deaa9edf-087d-40dc-b794-6e7d9280165d/volumes" Jul 22 01:46:52.788082 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:52.788031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:52.850220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:52.850175 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:54.854674 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:54.854617 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:54.956235 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:54.956144 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:46:59.983256 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:59.983228 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:46:59.987112 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:46:59.987088 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c9bf68595-vdlpl" Jul 22 01:47:00.752045 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:00.752012 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:00.853423 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:00.853391 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:02.227306 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.227258 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65769658cf-ht7s9" podUID="c239d27a-4d19-49f0-af31-6a344ed2e14b" containerName="console" containerID="cri-o://53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285" gracePeriod=15 Jul 22 01:47:02.481654 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.481566 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:47:02.529908 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.529876 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6tgc\" (UniqueName: \"kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.529950 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.529985 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530009 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530040 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530068 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530348 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530108 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca\") pod \"c239d27a-4d19-49f0-af31-6a344ed2e14b\" (UID: \"c239d27a-4d19-49f0-af31-6a344ed2e14b\") " Jul 22 01:47:02.530511 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530421 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config" (OuterVolumeSpecName: "console-config") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:02.530570 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530533 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:02.530654 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530608 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:02.530708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.530654 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca" (OuterVolumeSpecName: "service-ca") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:02.532332 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.532308 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:02.532437 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.532328 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc" (OuterVolumeSpecName: "kube-api-access-m6tgc") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "kube-api-access-m6tgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:47:02.532437 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.532344 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c239d27a-4d19-49f0-af31-6a344ed2e14b" (UID: "c239d27a-4d19-49f0-af31-6a344ed2e14b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:02.631739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631699 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631727 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631738 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631748 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631756 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c239d27a-4d19-49f0-af31-6a344ed2e14b-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631764 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c239d27a-4d19-49f0-af31-6a344ed2e14b-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:02.631999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:02.631772 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6tgc\" (UniqueName: \"kubernetes.io/projected/c239d27a-4d19-49f0-af31-6a344ed2e14b-kube-api-access-m6tgc\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:03.275268 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.275230 2585 generic.go:356] "Generic (PLEG): container finished" podID="c239d27a-4d19-49f0-af31-6a344ed2e14b" containerID="53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285" exitCode=0 Jul 22 01:47:03.275268 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.275272 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65769658cf-ht7s9" Jul 22 01:47:03.275796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.275294 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65769658cf-ht7s9" event={"ID":"c239d27a-4d19-49f0-af31-6a344ed2e14b","Type":"ContainerDied","Data":"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285"} Jul 22 01:47:03.275796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.275344 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65769658cf-ht7s9" event={"ID":"c239d27a-4d19-49f0-af31-6a344ed2e14b","Type":"ContainerDied","Data":"e867ff2d904d2cee97cb5a340902cc12c1638af0ad783fd3b3884d16af8b4422"} Jul 22 01:47:03.275796 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.275364 2585 scope.go:117] "RemoveContainer" containerID="53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285" Jul 22 01:47:03.283746 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.283727 2585 scope.go:117] "RemoveContainer" containerID="53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285" Jul 22 01:47:03.284073 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:03.284055 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285\": container with ID starting with 53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285 not found: ID does not exist" containerID="53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285" Jul 22 01:47:03.284151 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.284085 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285"} err="failed to get container status \"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285\": rpc error: code = NotFound desc = could not find container \"53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285\": container with ID starting with 53d3352d59b336c59699b10b4829d42e6a3fcbda5f4e293a9fc028d8a57a8285 not found: ID does not exist" Jul 22 01:47:03.293487 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.293458 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:47:03.298822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:03.298794 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65769658cf-ht7s9"] Jul 22 01:47:04.677338 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:04.677303 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c239d27a-4d19-49f0-af31-6a344ed2e14b" path="/var/lib/kubelet/pods/c239d27a-4d19-49f0-af31-6a344ed2e14b/volumes" Jul 22 01:47:05.000151 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000073 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:05.000422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000384 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="alertmanager" containerID="cri-o://e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f" gracePeriod=120 Jul 22 01:47:05.000422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000410 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-web" containerID="cri-o://172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad" gracePeriod=120 Jul 22 01:47:05.000618 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000444 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="prom-label-proxy" containerID="cri-o://fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8" gracePeriod=120 Jul 22 01:47:05.000618 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000470 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy" containerID="cri-o://73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072" gracePeriod=120 Jul 22 01:47:05.000618 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000416 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-metric" containerID="cri-o://32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e" gracePeriod=120 Jul 22 01:47:05.000618 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.000467 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="config-reloader" containerID="cri-o://167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825" gracePeriod=120 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288783 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8" exitCode=0 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288812 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e" exitCode=0 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288819 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072" exitCode=0 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288826 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825" exitCode=0 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288831 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f" exitCode=0 Jul 22 01:47:05.288878 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288853 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8"} Jul 22 01:47:05.289236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e"} Jul 22 01:47:05.289236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.288987 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072"} Jul 22 01:47:05.289236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.289011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825"} Jul 22 01:47:05.289236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:05.289033 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f"} Jul 22 01:47:06.143189 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.143161 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6bqbj" Jul 22 01:47:06.263894 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.263870 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.298413 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.298305 2585 generic.go:356] "Generic (PLEG): container finished" podID="b01f9309-05ee-401c-866c-0d31e80279d3" containerID="172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad" exitCode=0 Jul 22 01:47:06.298413 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.298368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad"} Jul 22 01:47:06.298413 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.298416 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.298735 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.298506 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b01f9309-05ee-401c-866c-0d31e80279d3","Type":"ContainerDied","Data":"a8ceabaac121dc371c7d487e15905ece1b98bee7f270c93a63d5c5c7447d8dad"} Jul 22 01:47:06.298735 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.298580 2585 scope.go:117] "RemoveContainer" containerID="fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8" Jul 22 01:47:06.308890 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.308847 2585 scope.go:117] "RemoveContainer" containerID="32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e" Jul 22 01:47:06.319185 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.319161 2585 scope.go:117] "RemoveContainer" containerID="73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072" Jul 22 01:47:06.328327 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.328307 2585 scope.go:117] "RemoveContainer" containerID="172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad" Jul 22 01:47:06.335578 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.335553 2585 scope.go:117] "RemoveContainer" containerID="167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825" Jul 22 01:47:06.344091 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.344071 2585 scope.go:117] "RemoveContainer" containerID="e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f" Jul 22 01:47:06.351600 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.351583 2585 scope.go:117] "RemoveContainer" containerID="a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd" Jul 22 01:47:06.358375 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.358350 2585 scope.go:117] "RemoveContainer" containerID="fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8" Jul 22 01:47:06.358696 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.358671 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8\": container with ID starting with fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8 not found: ID does not exist" containerID="fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8" Jul 22 01:47:06.358746 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.358709 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8"} err="failed to get container status \"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8\": rpc error: code = NotFound desc = could not find container \"fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8\": container with ID starting with fad25c8101b8546b522ac8a236f68c074fcb27340b87ed18acdc93db5e3beca8 not found: ID does not exist" Jul 22 01:47:06.358746 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.358729 2585 scope.go:117] "RemoveContainer" containerID="32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e" Jul 22 01:47:06.358952 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.358936 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e\": container with ID starting with 32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e not found: ID does not exist" containerID="32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e" Jul 22 01:47:06.358990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.358958 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e"} err="failed to get container status \"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e\": rpc error: code = NotFound desc = could not find container \"32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e\": container with ID starting with 32d2b922bc15306147d672a1b527038faaf44a466dfc99e3ab6cedb4767f754e not found: ID does not exist" Jul 22 01:47:06.358990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.358972 2585 scope.go:117] "RemoveContainer" containerID="73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072" Jul 22 01:47:06.359238 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.359219 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072\": container with ID starting with 73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072 not found: ID does not exist" containerID="73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072" Jul 22 01:47:06.359314 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359243 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072"} err="failed to get container status \"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072\": rpc error: code = NotFound desc = could not find container \"73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072\": container with ID starting with 73676ce7639c1600aefba6c19207dc71d7f957462ef948320e6374b1c78db072 not found: ID does not exist" Jul 22 01:47:06.359314 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359265 2585 scope.go:117] "RemoveContainer" containerID="172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad" Jul 22 01:47:06.359503 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.359487 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad\": container with ID starting with 172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad not found: ID does not exist" containerID="172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad" Jul 22 01:47:06.359542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359509 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad"} err="failed to get container status \"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad\": rpc error: code = NotFound desc = could not find container \"172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad\": container with ID starting with 172b623899696eb38cf3276d4e27a042d14dc767f81ffd6ffc24edab8408abad not found: ID does not exist" Jul 22 01:47:06.359542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359524 2585 scope.go:117] "RemoveContainer" containerID="167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825" Jul 22 01:47:06.359751 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.359734 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825\": container with ID starting with 167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825 not found: ID does not exist" containerID="167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825" Jul 22 01:47:06.359801 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359756 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825"} err="failed to get container status \"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825\": rpc error: code = NotFound desc = could not find container \"167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825\": container with ID starting with 167ea147f7b15e59758caaf4e8a94cbe2c9e7e67e2d47a04c72ede5208231825 not found: ID does not exist" Jul 22 01:47:06.359801 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359770 2585 scope.go:117] "RemoveContainer" containerID="e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f" Jul 22 01:47:06.359988 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.359972 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f\": container with ID starting with e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f not found: ID does not exist" containerID="e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f" Jul 22 01:47:06.360030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.359993 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f"} err="failed to get container status \"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f\": rpc error: code = NotFound desc = could not find container \"e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f\": container with ID starting with e4050112a807515ba0651e3a7276da55b9bb18bed1c177a5fc0025a067987b7f not found: ID does not exist" Jul 22 01:47:06.360030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.360006 2585 scope.go:117] "RemoveContainer" containerID="a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd" Jul 22 01:47:06.360194 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:06.360180 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd\": container with ID starting with a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd not found: ID does not exist" containerID="a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd" Jul 22 01:47:06.360238 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.360196 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd"} err="failed to get container status \"a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd\": rpc error: code = NotFound desc = could not find container \"a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd\": container with ID starting with a47c7cace5f4f4098667b3cb0e39e702b506a177e371e06a2f43fa7d431a78cd not found: ID does not exist" Jul 22 01:47:06.366520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366495 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366544 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366564 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366594 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366627 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366763 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366718 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366773 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366812 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366839 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqv7g\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.366910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366898 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.367098 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366921 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.367098 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out\") pod \"b01f9309-05ee-401c-866c-0d31e80279d3\" (UID: \"b01f9309-05ee-401c-866c-0d31e80279d3\") " Jul 22 01:47:06.367505 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.366945 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:06.367505 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.367143 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:06.367505 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.367233 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:47:06.370094 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370063 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume" (OuterVolumeSpecName: "config-volume") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.370415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370371 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out" (OuterVolumeSpecName: "config-out") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:47:06.370522 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370464 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.370522 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370495 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:47:06.370696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370585 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.370696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370655 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g" (OuterVolumeSpecName: "kube-api-access-cqv7g") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "kube-api-access-cqv7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:47:06.370817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370711 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.370817 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.370798 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.374861 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.374697 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.381764 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.381735 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config" (OuterVolumeSpecName: "web-config") pod "b01f9309-05ee-401c-866c-0d31e80279d3" (UID: "b01f9309-05ee-401c-866c-0d31e80279d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:06.468350 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468301 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468350 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468347 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468350 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468360 2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-config-volume\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468350 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468370 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468380 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-alertmanager-main-db\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468388 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468398 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b01f9309-05ee-401c-866c-0d31e80279d3-metrics-client-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468407 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-secret-alertmanager-main-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468416 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-web-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468424 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqv7g\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-kube-api-access-cqv7g\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468431 2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b01f9309-05ee-401c-866c-0d31e80279d3-cluster-tls-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468439 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b01f9309-05ee-401c-866c-0d31e80279d3-tls-assets\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.468614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.468449 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b01f9309-05ee-401c-866c-0d31e80279d3-config-out\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:06.623837 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.623796 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:06.627677 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.627627 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:06.654154 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654121 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:06.654431 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654418 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy" Jul 22 01:47:06.654472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654432 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy" Jul 22 01:47:06.654472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654443 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c239d27a-4d19-49f0-af31-6a344ed2e14b" containerName="console" Jul 22 01:47:06.654472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654449 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c239d27a-4d19-49f0-af31-6a344ed2e14b" containerName="console" Jul 22 01:47:06.654472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654461 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="config-reloader" Jul 22 01:47:06.654472 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654467 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="config-reloader" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654477 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45141ee-5c79-418c-b252-b6ca59be6cb3" containerName="console" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654482 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45141ee-5c79-418c-b252-b6ca59be6cb3" containerName="console" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654492 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="init-config-reloader" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654498 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="init-config-reloader" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654504 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deaa9edf-087d-40dc-b794-6e7d9280165d" containerName="console" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654509 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="deaa9edf-087d-40dc-b794-6e7d9280165d" containerName="console" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654518 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-web" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654523 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-web" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654530 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="prom-label-proxy" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654535 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="prom-label-proxy" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654542 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="alertmanager" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654548 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="alertmanager" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654554 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-metric" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654558 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-metric" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654603 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c239d27a-4d19-49f0-af31-6a344ed2e14b" containerName="console" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654612 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654618 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="config-reloader" Jul 22 01:47:06.654616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654624 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="deaa9edf-087d-40dc-b794-6e7d9280165d" containerName="console" Jul 22 01:47:06.655389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654630 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-metric" Jul 22 01:47:06.655389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654648 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="prom-label-proxy" Jul 22 01:47:06.655389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654653 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45141ee-5c79-418c-b252-b6ca59be6cb3" containerName="console" Jul 22 01:47:06.655389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654658 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="alertmanager" Jul 22 01:47:06.655389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.654666 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" containerName="kube-rbac-proxy-web" Jul 22 01:47:06.658795 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.658775 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.661529 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.661506 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 01:47:06.661647 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.661562 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8m9hd\"" Jul 22 01:47:06.661991 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.661963 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 01:47:06.662107 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662041 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 01:47:06.662107 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662063 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 01:47:06.662220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662118 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 01:47:06.662220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662197 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 01:47:06.662321 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662271 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 01:47:06.662321 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.662287 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 01:47:06.667147 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.667129 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 01:47:06.677186 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.677159 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01f9309-05ee-401c-866c-0d31e80279d3" path="/var/lib/kubelet/pods/b01f9309-05ee-401c-866c-0d31e80279d3/volumes" Jul 22 01:47:06.677697 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.677683 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:06.771658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-config-out\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771756 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrgb\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-kube-api-access-lzrgb\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-web-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.771966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.772054 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771964 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.772054 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.771987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.772054 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.772003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.772054 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.772022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.772168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.772069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.872929 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.872827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrgb\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-kube-api-access-lzrgb\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.872929 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.872886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-web-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.872929 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.872916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.872944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.872981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-config-out\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873686 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873240 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.873686 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.873278 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.875163 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.874226 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.875163 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.874871 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d256af7e-8364-4946-8e2a-09563ea920d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876072 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.875934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876072 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.875982 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876086 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876389 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876457 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876689 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876667 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.876772 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.876755 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-web-config\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.877183 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.877165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d256af7e-8364-4946-8e2a-09563ea920d8-config-out\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.878060 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.878040 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d256af7e-8364-4946-8e2a-09563ea920d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.881196 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.881178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrgb\" (UniqueName: \"kubernetes.io/projected/d256af7e-8364-4946-8e2a-09563ea920d8-kube-api-access-lzrgb\") pod \"alertmanager-main-0\" (UID: \"d256af7e-8364-4946-8e2a-09563ea920d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:06.969064 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:06.969030 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:47:07.116494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:07.116399 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:47:07.118491 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:47:07.118461 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd256af7e_8364_4946_8e2a_09563ea920d8.slice/crio-1aab648dc62c9bfcaf5f58a7088a9ecba2cccd913adc4fa8d059571bfe98a7e9 WatchSource:0}: Error finding container 1aab648dc62c9bfcaf5f58a7088a9ecba2cccd913adc4fa8d059571bfe98a7e9: Status 404 returned error can't find the container with id 1aab648dc62c9bfcaf5f58a7088a9ecba2cccd913adc4fa8d059571bfe98a7e9 Jul 22 01:47:07.305854 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:07.305818 2585 generic.go:356] "Generic (PLEG): container finished" podID="d256af7e-8364-4946-8e2a-09563ea920d8" containerID="abe0ae477ecc196105a3f2677f43865fe67014a201eb9a9e56f04bd90879d1b8" exitCode=0 Jul 22 01:47:07.306336 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:07.305896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerDied","Data":"abe0ae477ecc196105a3f2677f43865fe67014a201eb9a9e56f04bd90879d1b8"} Jul 22 01:47:07.306336 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:07.306008 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"1aab648dc62c9bfcaf5f58a7088a9ecba2cccd913adc4fa8d059571bfe98a7e9"} Jul 22 01:47:08.315275 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"c4df31489ed60b92e6bf8c22869234ebf51cbc6c28b00bc3ff63ce92a2b41e64"} Jul 22 01:47:08.315757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315303 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"fb8d3978e99dca899495af8550316da4589f793f4d34a2077a2c6590ddcd33da"} Jul 22 01:47:08.315757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"b12dd0084cf04842d2c0d7799876e4adfc3ce55e9ebf0c6b81e7bb6c7882147b"} Jul 22 01:47:08.315757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315355 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"3b3b9fcf220226d14392f5e48df36003cd32ebb5afd4916f0d810cc5a5112a1f"} Jul 22 01:47:08.315757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"426c861a7f5462b278fae089c84bbabe68db660a02e27c0a389be5bc09ae8d83"} Jul 22 01:47:08.315757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.315401 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d256af7e-8364-4946-8e2a-09563ea920d8","Type":"ContainerStarted","Data":"da6b06ac7494498865282f4721fb23359fb46bb72e881ebaaa3f91812d4c3074"} Jul 22 01:47:08.342768 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:08.342627 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.327157747 podStartE2EDuration="2.342608427s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.45068ms" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:08.342157686 +0000 UTC m=+106.243036187" watchObservedRunningTime="2026-07-22 01:47:08.342608427 +0000 UTC m=+106.243486924" Jul 22 01:47:09.243956 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.243913 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:09.244404 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244366 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="prometheus" containerID="cri-o://7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5" gracePeriod=600 Jul 22 01:47:09.244631 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244373 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="config-reloader" containerID="cri-o://321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027" gracePeriod=600 Jul 22 01:47:09.244773 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244372 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="thanos-sidecar" containerID="cri-o://84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab" gracePeriod=600 Jul 22 01:47:09.244773 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244399 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274" gracePeriod=600 Jul 22 01:47:09.244849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244407 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy" containerID="cri-o://37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d" gracePeriod=600 Jul 22 01:47:09.244921 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:09.244408 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-web" containerID="cri-o://d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1" gracePeriod=600 Jul 22 01:47:10.330888 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330861 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274" exitCode=0 Jul 22 01:47:10.330888 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330885 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d" exitCode=0 Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330893 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab" exitCode=0 Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330899 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027" exitCode=0 Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330904 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5" exitCode=0 Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330927 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274"} Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.330987 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d"} Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.331013 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab"} Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.331037 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027"} Jul 22 01:47:10.331208 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.331065 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5"} Jul 22 01:47:10.496047 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.496021 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:10.606913 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606813 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.606913 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606861 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.606913 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606925 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606949 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.606974 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607011 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607048 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607083 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607111 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607147 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607179 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607214 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607245 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607278 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607326 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607348 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607368 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607396 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607396 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607424 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.607494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607447 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr6kd\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd\") pod \"ed415836-0d1b-4276-b743-d9e4cd714eb7\" (UID: \"ed415836-0d1b-4276-b743-d9e4cd714eb7\") " Jul 22 01:47:10.608073 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607761 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-metrics-client-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.608073 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607780 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.608073 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.607922 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:47:10.608244 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.608219 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.608867 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.608501 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.608867 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.608613 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.610317 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610054 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.610460 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610388 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:47:10.610460 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610429 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.610598 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610494 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config" (OuterVolumeSpecName: "config") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.610789 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610761 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.610859 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610820 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:47:10.610922 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.610902 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.611031 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.611012 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd" (OuterVolumeSpecName: "kube-api-access-dr6kd") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "kube-api-access-dr6kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:47:10.611100 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.611063 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.611185 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.611163 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.612573 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.612545 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.612801 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.612680 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.612889 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.612794 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out" (OuterVolumeSpecName: "config-out") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:47:10.623444 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.623412 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config" (OuterVolumeSpecName: "web-config") pod "ed415836-0d1b-4276-b743-d9e4cd714eb7" (UID: "ed415836-0d1b-4276-b743-d9e4cd714eb7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:47:10.709025 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.708982 2585 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-grpc-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709025 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709016 2585 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709025 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709029 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709044 2585 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-metrics-client-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709056 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709065 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709075 2585 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-kube-rbac-proxy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709085 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-web-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709094 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-config-out\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709103 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6kd\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-kube-api-access-dr6kd\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709117 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed415836-0d1b-4276-b743-d9e4cd714eb7-tls-assets\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709126 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709133 2585 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709141 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709150 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-db\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709158 2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed415836-0d1b-4276-b743-d9e4cd714eb7-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709166 2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:10.709250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:10.709176 2585 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed415836-0d1b-4276-b743-d9e4cd714eb7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:47:11.341676 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.341628 2585 generic.go:356] "Generic (PLEG): container finished" podID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerID="d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1" exitCode=0 Jul 22 01:47:11.342047 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.341678 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1"} Jul 22 01:47:11.342047 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.341743 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.342047 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.341769 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed415836-0d1b-4276-b743-d9e4cd714eb7","Type":"ContainerDied","Data":"85794917120bf7bd30a783241c08f20ec65773717deae266c1ad6b479cbb900b"} Jul 22 01:47:11.342047 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.341817 2585 scope.go:117] "RemoveContainer" containerID="8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274" Jul 22 01:47:11.350333 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.350309 2585 scope.go:117] "RemoveContainer" containerID="37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d" Jul 22 01:47:11.358170 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.357987 2585 scope.go:117] "RemoveContainer" containerID="d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1" Jul 22 01:47:11.365792 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.365766 2585 scope.go:117] "RemoveContainer" containerID="84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab" Jul 22 01:47:11.366699 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.366670 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:11.371623 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.371595 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:11.374153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.374137 2585 scope.go:117] "RemoveContainer" containerID="321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027" Jul 22 01:47:11.386865 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.386841 2585 scope.go:117] "RemoveContainer" containerID="7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5" Jul 22 01:47:11.395394 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.395371 2585 scope.go:117] "RemoveContainer" containerID="5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43" Jul 22 01:47:11.395693 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.395673 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:11.396106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396091 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="prometheus" Jul 22 01:47:11.396168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396108 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="prometheus" Jul 22 01:47:11.396168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396128 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="thanos-sidecar" Jul 22 01:47:11.396168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396137 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="thanos-sidecar" Jul 22 01:47:11.396168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396147 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-web" Jul 22 01:47:11.396168 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396156 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-web" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396171 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="config-reloader" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396180 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="config-reloader" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396187 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396192 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396204 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="init-config-reloader" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396209 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="init-config-reloader" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396215 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-thanos" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396220 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-thanos" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396268 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="thanos-sidecar" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396276 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="prometheus" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396291 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-web" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396297 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy-thanos" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396303 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="config-reloader" Jul 22 01:47:11.396363 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.396309 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" containerName="kube-rbac-proxy" Jul 22 01:47:11.400388 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.400369 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.403617 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.403590 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 01:47:11.404365 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.403878 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2irkv8o4kpl\"" Jul 22 01:47:11.404365 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.403915 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 01:47:11.404518 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.404385 2585 scope.go:117] "RemoveContainer" containerID="8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274" Jul 22 01:47:11.404908 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.404884 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 01:47:11.405216 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.405020 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274\": container with ID starting with 8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274 not found: ID does not exist" containerID="8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274" Jul 22 01:47:11.405216 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.405067 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274"} err="failed to get container status \"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274\": rpc error: code = NotFound desc = could not find container \"8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274\": container with ID starting with 8ebc4382665a32f7736bff22fa736b44efa7571656dc37f08ddb867c1a70c274 not found: ID does not exist" Jul 22 01:47:11.405216 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.405095 2585 scope.go:117] "RemoveContainer" containerID="37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d" Jul 22 01:47:11.405448 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.405251 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.405759 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.406123 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.406761 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.407086 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.407320 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.407609 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.407978 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z4wf8\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.408268 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.408556 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.410444 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d\": container with ID starting with 37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d not found: ID does not exist" containerID="37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.410933 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d"} err="failed to get container status \"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d\": rpc error: code = NotFound desc = could not find container \"37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d\": container with ID starting with 37407062fc1addea0fa8f63d9709a8df2d8a6eb752865f03494784610b2e375d not found: ID does not exist" Jul 22 01:47:11.411005 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.410980 2585 scope.go:117] "RemoveContainer" containerID="d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1" Jul 22 01:47:11.414975 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.414943 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1\": container with ID starting with d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1 not found: ID does not exist" containerID="d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1" Jul 22 01:47:11.415116 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.414986 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1"} err="failed to get container status \"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1\": rpc error: code = NotFound desc = could not find container \"d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1\": container with ID starting with d8591d991d18206aee586a5c67c33329f88d346660e56d8bd6bf6b141dfb21d1 not found: ID does not exist" Jul 22 01:47:11.415116 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.415012 2585 scope.go:117] "RemoveContainer" containerID="84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab" Jul 22 01:47:11.415395 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.415364 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab\": container with ID starting with 84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab not found: ID does not exist" containerID="84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab" Jul 22 01:47:11.415515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.415398 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab"} err="failed to get container status \"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab\": rpc error: code = NotFound desc = could not find container \"84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab\": container with ID starting with 84a163318f06bd377344bb2188760e69b717d0e72ed2138bbd51f9d0f72b64ab not found: ID does not exist" Jul 22 01:47:11.415515 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.415420 2585 scope.go:117] "RemoveContainer" containerID="321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027" Jul 22 01:47:11.415973 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.415847 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027\": container with ID starting with 321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027 not found: ID does not exist" containerID="321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027" Jul 22 01:47:11.415973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.415879 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027"} err="failed to get container status \"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027\": rpc error: code = NotFound desc = could not find container \"321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027\": container with ID starting with 321819eb2cc8e589fa9789724895a6ac15d935986273e7b527ca7faf6ef73027 not found: ID does not exist" Jul 22 01:47:11.415973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.415899 2585 scope.go:117] "RemoveContainer" containerID="7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5" Jul 22 01:47:11.416306 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.416282 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 01:47:11.416492 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.416466 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5\": container with ID starting with 7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5 not found: ID does not exist" containerID="7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5" Jul 22 01:47:11.416562 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.416500 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5"} err="failed to get container status \"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5\": rpc error: code = NotFound desc = could not find container \"7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5\": container with ID starting with 7aacddbf49e0a4227f053b26b2aed701533973c4676ed18739aabe83ddb034e5 not found: ID does not exist" Jul 22 01:47:11.416562 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.416521 2585 scope.go:117] "RemoveContainer" containerID="5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43" Jul 22 01:47:11.416904 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:47:11.416872 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43\": container with ID starting with 5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43 not found: ID does not exist" containerID="5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43" Jul 22 01:47:11.416973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.416912 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43"} err="failed to get container status \"5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43\": rpc error: code = NotFound desc = could not find container \"5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43\": container with ID starting with 5e3fbbb55b44dc4de63c2b1b2745334060cfc43272334179c5898ac24c7e2d43 not found: ID does not exist" Jul 22 01:47:11.417835 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.417816 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:11.417922 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.417907 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 01:47:11.516307 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516307 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516340 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516413 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lbn\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-kube-api-access-h2lbn\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516525 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516545 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516683 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516705 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516735 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.516805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.517075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.517075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.516832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617428 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lbn\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-kube-api-access-h2lbn\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617863 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617863 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617863 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.617863 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.617964 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618186 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618253 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618266 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618280 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618328 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618545 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.618988 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.618903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.619608 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.619585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.620036 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.620016 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.620577 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.620555 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.620687 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.620586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.621286 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.621265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.621381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.621287 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45600ff1-4e85-4511-a211-60a23e8f13c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.621479 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.621390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.621832 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.621810 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.621970 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.621890 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45600ff1-4e85-4511-a211-60a23e8f13c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.622153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.622129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.622229 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.622206 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.623202 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.623175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.623298 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.623263 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.623730 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.623711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.624118 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.624093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.624283 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.624263 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45600ff1-4e85-4511-a211-60a23e8f13c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.625862 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.625843 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lbn\" (UniqueName: \"kubernetes.io/projected/45600ff1-4e85-4511-a211-60a23e8f13c0-kube-api-access-h2lbn\") pod \"prometheus-k8s-0\" (UID: \"45600ff1-4e85-4511-a211-60a23e8f13c0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.717068 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.717026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:11.859534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:11.859427 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 01:47:11.862060 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:47:11.862037 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45600ff1_4e85_4511_a211_60a23e8f13c0.slice/crio-38abd212219bfd72ea9146559d2360636bd678a3c1b6560d007486bad7f83b12 WatchSource:0}: Error finding container 38abd212219bfd72ea9146559d2360636bd678a3c1b6560d007486bad7f83b12: Status 404 returned error can't find the container with id 38abd212219bfd72ea9146559d2360636bd678a3c1b6560d007486bad7f83b12 Jul 22 01:47:12.347992 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.347958 2585 generic.go:356] "Generic (PLEG): container finished" podID="45600ff1-4e85-4511-a211-60a23e8f13c0" containerID="4a1ce18467b31bd63d0933b049313458c9665634d249a1cb8d1678ceb5408735" exitCode=0 Jul 22 01:47:12.348421 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.348037 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerDied","Data":"4a1ce18467b31bd63d0933b049313458c9665634d249a1cb8d1678ceb5408735"} Jul 22 01:47:12.348421 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.348105 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"38abd212219bfd72ea9146559d2360636bd678a3c1b6560d007486bad7f83b12"} Jul 22 01:47:12.648963 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.648933 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:12.682666 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.679667 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed415836-0d1b-4276-b743-d9e4cd714eb7" path="/var/lib/kubelet/pods/ed415836-0d1b-4276-b743-d9e4cd714eb7/volumes" Jul 22 01:47:12.748584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:12.748533 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:13.356441 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356404 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"09e3bce4db1fbedf0e3ea600394bd276fd2b1b428a87b7ca156b8816d1281539"} Jul 22 01:47:13.356833 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356468 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"065f20d62bc634da0994227be3fb9ce44ca435a35c69020eae16f911c6d0e5bb"} Jul 22 01:47:13.356833 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356494 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"a190adf27032a17c2451edb7e75d0684f2f13187c4791c951900f885ab36906f"} Jul 22 01:47:13.356833 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"f7e4d7ca71a02659ccca70e01280c209d33a1446f3bdb1ff86d76477da53cac3"} Jul 22 01:47:13.356833 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356543 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"da1ba49e7f900ad5ec23f2237486929331953dd6106d0a79fecc83454b6955f1"} Jul 22 01:47:13.356833 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.356565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45600ff1-4e85-4511-a211-60a23e8f13c0","Type":"ContainerStarted","Data":"c9ec1cec41f233e643bbca2a51ba1a61554f81a463d12354fd211a5f8098602e"} Jul 22 01:47:13.387387 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:13.387317 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.369293803 podStartE2EDuration="2.387293597s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.999794ms" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:13.384463954 +0000 UTC m=+111.285342482" watchObservedRunningTime="2026-07-22 01:47:13.387293597 +0000 UTC m=+111.288172095" Jul 22 01:47:16.718122 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:16.718076 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:47:22.800587 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:22.800499 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:22.901966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:22.901937 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:34.799546 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.799505 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:47:34.804805 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.804781 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.813193 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.813170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:47:34.834542 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834743 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4qm\" (UniqueName: \"kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834743 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834743 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834743 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834905 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.834905 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.834787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.864366 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.864332 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:34.917868 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.917838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:34.936176 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936143 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4qm\" (UniqueName: \"kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936176 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936180 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936377 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936377 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936377 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936407 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.936525 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.936455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.937065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.937040 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.937065 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.937070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.937237 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.937216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.937478 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.937455 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.938735 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.938715 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.938851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.938835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:34.944289 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:34.944267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4qm\" (UniqueName: \"kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm\") pod \"console-8c4ccb47-d84cg\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:35.000200 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.000166 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:35.060399 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.060313 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:35.115082 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.115032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:35.250585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.250526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:47:35.252624 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:47:35.252595 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140f31c2_3023_431b_bee0_02b843e7c1a9.slice/crio-9b414f4c405d840a72cab5dfefe0a51566d095a66e282867a5f4e7f745817e01 WatchSource:0}: Error finding container 9b414f4c405d840a72cab5dfefe0a51566d095a66e282867a5f4e7f745817e01: Status 404 returned error can't find the container with id 9b414f4c405d840a72cab5dfefe0a51566d095a66e282867a5f4e7f745817e01 Jul 22 01:47:35.462453 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.462415 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c4ccb47-d84cg" event={"ID":"140f31c2-3023-431b-bee0-02b843e7c1a9","Type":"ContainerStarted","Data":"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845"} Jul 22 01:47:35.462688 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.462488 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c4ccb47-d84cg" event={"ID":"140f31c2-3023-431b-bee0-02b843e7c1a9","Type":"ContainerStarted","Data":"9b414f4c405d840a72cab5dfefe0a51566d095a66e282867a5f4e7f745817e01"} Jul 22 01:47:35.482274 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.482221 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c4ccb47-d84cg" podStartSLOduration=1.482204811 podStartE2EDuration="1.482204811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:35.479758495 +0000 UTC m=+133.380636990" watchObservedRunningTime="2026-07-22 01:47:35.482204811 +0000 UTC m=+133.383083306" Jul 22 01:47:35.619035 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.618943 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:35.678851 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:35.678815 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:45.115494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.115447 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:45.115983 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.115534 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:45.120192 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.120168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:45.505524 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.505494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:47:45.551316 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.551281 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:47:45.646657 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.646616 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:45.703408 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.703372 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:45.758443 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:45.758342 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:47:47.056802 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:47:47.056767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:10.578106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.578048 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7966975879-vfkf6" podUID="09d82c40-7f76-4732-ad94-70c74939c7e2" containerName="console" containerID="cri-o://321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988" gracePeriod=15 Jul 22 01:48:10.822229 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.822201 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:48:10.873993 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.873892 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.873993 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.873940 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874085 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874136 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ghhw\" (UniqueName: \"kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874184 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874211 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert\") pod \"09d82c40-7f76-4732-ad94-70c74939c7e2\" (UID: \"09d82c40-7f76-4732-ad94-70c74939c7e2\") " Jul 22 01:48:10.874477 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874331 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:48:10.874477 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874358 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:48:10.874575 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874494 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:48:10.874663 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874614 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config" (OuterVolumeSpecName: "console-config") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:48:10.874725 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874698 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.874725 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874716 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.874797 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.874729 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.876311 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.876277 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw" (OuterVolumeSpecName: "kube-api-access-8ghhw") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "kube-api-access-8ghhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:48:10.876422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.876288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:48:10.876422 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.876369 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09d82c40-7f76-4732-ad94-70c74939c7e2" (UID: "09d82c40-7f76-4732-ad94-70c74939c7e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:48:10.975480 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.975441 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghhw\" (UniqueName: \"kubernetes.io/projected/09d82c40-7f76-4732-ad94-70c74939c7e2-kube-api-access-8ghhw\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.975480 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.975469 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09d82c40-7f76-4732-ad94-70c74939c7e2-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.975721 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.975501 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:10.975721 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:10.975511 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09d82c40-7f76-4732-ad94-70c74939c7e2-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:48:11.608072 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.608039 2585 generic.go:356] "Generic (PLEG): container finished" podID="09d82c40-7f76-4732-ad94-70c74939c7e2" containerID="321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988" exitCode=0 Jul 22 01:48:11.608468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.608079 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7966975879-vfkf6" Jul 22 01:48:11.608468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.608118 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966975879-vfkf6" event={"ID":"09d82c40-7f76-4732-ad94-70c74939c7e2","Type":"ContainerDied","Data":"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988"} Jul 22 01:48:11.608468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.608197 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7966975879-vfkf6" event={"ID":"09d82c40-7f76-4732-ad94-70c74939c7e2","Type":"ContainerDied","Data":"7a319bb1b598e1e61302265709de31350f9ca77496a0decd17be06597e1c32f5"} Jul 22 01:48:11.608468 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.608220 2585 scope.go:117] "RemoveContainer" containerID="321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988" Jul 22 01:48:11.616965 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.616945 2585 scope.go:117] "RemoveContainer" containerID="321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988" Jul 22 01:48:11.617263 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:48:11.617245 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988\": container with ID starting with 321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988 not found: ID does not exist" containerID="321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988" Jul 22 01:48:11.617309 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.617275 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988"} err="failed to get container status \"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988\": rpc error: code = NotFound desc = could not find container \"321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988\": container with ID starting with 321f201499b73c79dfeda10f30669923c09db78a27fadb2cbb6ce425c20cd988 not found: ID does not exist" Jul 22 01:48:11.628347 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.628315 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:48:11.631167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.631144 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7966975879-vfkf6"] Jul 22 01:48:11.717814 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.717762 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:48:11.733482 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:11.733457 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:48:12.628317 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:12.628292 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 01:48:12.682462 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:12.682425 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d82c40-7f76-4732-ad94-70c74939c7e2" path="/var/lib/kubelet/pods/09d82c40-7f76-4732-ad94-70c74939c7e2/volumes" Jul 22 01:48:14.448773 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:14.448731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:14.503938 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:14.503906 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:15.396128 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:15.396083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:15.497737 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:15.497701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:18.257499 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:18.257432 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:42.106187 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:42.106146 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:42.884341 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:42.884305 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:45.721095 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:45.721048 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:45.820486 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:45.820451 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:47.569222 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:47.569178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:47.685923 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:47.685888 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:47.992125 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:47.992090 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:48.091046 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:48.091011 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:50.380007 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:50.379969 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:50.782887 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:50.782850 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:52.715369 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:52.715331 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:52.815658 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:52.815592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:56.080052 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.080013 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xx7mz"] Jul 22 01:48:56.080483 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.080390 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d82c40-7f76-4732-ad94-70c74939c7e2" containerName="console" Jul 22 01:48:56.080483 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.080402 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d82c40-7f76-4732-ad94-70c74939c7e2" containerName="console" Jul 22 01:48:56.080567 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.080495 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d82c40-7f76-4732-ad94-70c74939c7e2" containerName="console" Jul 22 01:48:56.083501 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.083482 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.086079 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.086056 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vzbr2\"" Jul 22 01:48:56.086550 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.086531 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 01:48:56.087338 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.087322 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 01:48:56.093474 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.093444 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xx7mz"] Jul 22 01:48:56.185122 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.185083 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnxs\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-kube-api-access-sbnxs\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.185319 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.185147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.285986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.285947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnxs\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-kube-api-access-sbnxs\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.286142 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.286027 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.294990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.294959 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.295250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.295227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnxs\" (UniqueName: \"kubernetes.io/projected/7fed019f-9938-48c2-a6e9-73978e15c4c1-kube-api-access-sbnxs\") pod \"cert-manager-webhook-6486ffd997-xx7mz\" (UID: \"7fed019f-9938-48c2-a6e9-73978e15c4c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.410821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.410720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:48:56.542652 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.542600 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xx7mz"] Jul 22 01:48:56.546402 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:48:56.546377 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fed019f_9938_48c2_a6e9_73978e15c4c1.slice/crio-16c20a2d61b87e4e1d049068961d301e3e79717097bd6e3153007c6e37284d7c WatchSource:0}: Error finding container 16c20a2d61b87e4e1d049068961d301e3e79717097bd6e3153007c6e37284d7c: Status 404 returned error can't find the container with id 16c20a2d61b87e4e1d049068961d301e3e79717097bd6e3153007c6e37284d7c Jul 22 01:48:56.788041 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:56.788011 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" event={"ID":"7fed019f-9938-48c2-a6e9-73978e15c4c1","Type":"ContainerStarted","Data":"16c20a2d61b87e4e1d049068961d301e3e79717097bd6e3153007c6e37284d7c"} Jul 22 01:48:57.986359 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:57.986309 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:58.280399 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.280313 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwgs8"] Jul 22 01:48:58.289557 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.289480 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.292445 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.292169 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c6x7m\"" Jul 22 01:48:58.293684 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.293607 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwgs8"] Jul 22 01:48:58.304597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.304565 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrfx\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-kube-api-access-wrrfx\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.304795 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.304611 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.405482 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.405439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.405706 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.405686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrfx\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-kube-api-access-wrrfx\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.415041 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.415014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.415912 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.415884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrfx\" (UniqueName: \"kubernetes.io/projected/a8806277-e2c5-4c9a-b899-cf59cbea70e1-kube-api-access-wrrfx\") pod \"cert-manager-55b4d954d5-gwgs8\" (UID: \"a8806277-e2c5-4c9a-b899-cf59cbea70e1\") " pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:58.558330 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.558252 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:48:58.603353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:58.603317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gwgs8" Jul 22 01:48:59.797729 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:48:59.797703 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwgs8"] Jul 22 01:48:59.799927 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:48:59.799899 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8806277_e2c5_4c9a_b899_cf59cbea70e1.slice/crio-6df7187adbecc805cbda526e30046359007c82828c9c6ac52f120c69cff643d0 WatchSource:0}: Error finding container 6df7187adbecc805cbda526e30046359007c82828c9c6ac52f120c69cff643d0: Status 404 returned error can't find the container with id 6df7187adbecc805cbda526e30046359007c82828c9c6ac52f120c69cff643d0 Jul 22 01:49:00.218296 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.218235 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:00.312853 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.312819 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:00.808343 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.808300 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gwgs8" event={"ID":"a8806277-e2c5-4c9a-b899-cf59cbea70e1","Type":"ContainerStarted","Data":"d545d3f2e2d8ebe37921df0ed3b9faae163927bb390afe5d8e2151db7a6287ac"} Jul 22 01:49:00.808822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.808350 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gwgs8" event={"ID":"a8806277-e2c5-4c9a-b899-cf59cbea70e1","Type":"ContainerStarted","Data":"6df7187adbecc805cbda526e30046359007c82828c9c6ac52f120c69cff643d0"} Jul 22 01:49:00.809846 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.809824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" event={"ID":"7fed019f-9938-48c2-a6e9-73978e15c4c1","Type":"ContainerStarted","Data":"dc42e7ada4dfbfc51e3fe7446e4f1e10aee621d46850a33812c3e5be9b30e8f9"} Jul 22 01:49:00.809978 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.809961 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:49:00.823322 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.823063 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gwgs8" podStartSLOduration=2.8230442 podStartE2EDuration="2.8230442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:00.822909292 +0000 UTC m=+218.723787810" watchObservedRunningTime="2026-07-22 01:49:00.8230442 +0000 UTC m=+218.723922699" Jul 22 01:49:00.838618 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:00.838552 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" podStartSLOduration=1.652645454 podStartE2EDuration="4.838528453s" totalImagesPullingTime="3.185882999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:00.837809312 +0000 UTC m=+218.738687833" watchObservedRunningTime="2026-07-22 01:49:00.838528453 +0000 UTC m=+218.739406952" Jul 22 01:49:06.009954 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:06.009920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:06.111409 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:06.111376 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:06.816433 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:06.816406 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xx7mz" Jul 22 01:49:07.208219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:07.208186 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:07.312605 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:07.312569 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:07.676829 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:07.676795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.160820 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:24.160779 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.265911 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:24.265860 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.773994 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:24.773956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:24.977002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:24.976939 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.093234 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.093121 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.177009 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.176973 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp"] Jul 22 01:49:25.180995 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.180977 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.185993 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.185972 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 01:49:25.186415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.186396 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 01:49:25.186504 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.186427 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 01:49:25.186504 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.186444 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 01:49:25.186836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.186818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ghhvt\"" Jul 22 01:49:25.200780 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.200743 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp"] Jul 22 01:49:25.328934 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.328903 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.367891 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.367808 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.367891 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.367847 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.368077 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.367931 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.368077 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.367965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bj8g\" (UniqueName: \"kubernetes.io/projected/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-kube-api-access-5bj8g\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.368077 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.367984 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.429320 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.429292 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.469203 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.469203 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469207 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.469431 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.469431 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469283 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bj8g\" (UniqueName: \"kubernetes.io/projected/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-kube-api-access-5bj8g\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.469431 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.469707 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.469685 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.471755 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.471730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.471865 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.471820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.471865 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.471849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.479585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.479557 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bj8g\" (UniqueName: \"kubernetes.io/projected/7b92e3ab-dcb0-467a-8c8d-f472021b34e3-kube-api-access-5bj8g\") pod \"opendatahub-operator-controller-manager-585784b644-mptdp\" (UID: \"7b92e3ab-dcb0-467a-8c8d-f472021b34e3\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.491302 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.491272 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:25.530476 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.530444 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.606101 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.606066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.635294 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.635268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp"] Jul 22 01:49:25.637303 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:49:25.637280 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b92e3ab_dcb0_467a_8c8d_f472021b34e3.slice/crio-711d3f836e7e7fe6be58ab7090e0da410766254d7a39b3cc19718d929743cfad WatchSource:0}: Error finding container 711d3f836e7e7fe6be58ab7090e0da410766254d7a39b3cc19718d929743cfad: Status 404 returned error can't find the container with id 711d3f836e7e7fe6be58ab7090e0da410766254d7a39b3cc19718d929743cfad Jul 22 01:49:25.670450 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.670417 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.803823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.803782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.881955 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.881920 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:25.911949 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:25.911857 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" event={"ID":"7b92e3ab-dcb0-467a-8c8d-f472021b34e3","Type":"ContainerStarted","Data":"711d3f836e7e7fe6be58ab7090e0da410766254d7a39b3cc19718d929743cfad"} Jul 22 01:49:28.929239 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:28.929209 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" event={"ID":"7b92e3ab-dcb0-467a-8c8d-f472021b34e3","Type":"ContainerStarted","Data":"6d08af78c73d092a0a64fa22294f1ae2fb34cd773ae9f83c045cd71023875be4"} Jul 22 01:49:29.920985 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:29.920952 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:29.935244 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:29.935212 2585 generic.go:356] "Generic (PLEG): container finished" podID="7b92e3ab-dcb0-467a-8c8d-f472021b34e3" containerID="6d08af78c73d092a0a64fa22294f1ae2fb34cd773ae9f83c045cd71023875be4" exitCode=0 Jul 22 01:49:29.935621 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:29.935306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" event={"ID":"7b92e3ab-dcb0-467a-8c8d-f472021b34e3","Type":"ContainerDied","Data":"6d08af78c73d092a0a64fa22294f1ae2fb34cd773ae9f83c045cd71023875be4"} Jul 22 01:49:30.024612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:30.024577 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:30.942546 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:30.942502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" event={"ID":"7b92e3ab-dcb0-467a-8c8d-f472021b34e3","Type":"ContainerStarted","Data":"9c6da968eb09499a82fd2d65fb184c99f8e093d14de5fb05a6b12a0ae033b21b"} Jul 22 01:49:30.942980 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:30.942661 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:30.972648 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:30.972587 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" podStartSLOduration=1.6998522569999999 podStartE2EDuration="5.97256368s" totalImagesPullingTime="3.227419075s" totalInitContainerRuntime="1.045292348s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:30.971171242 +0000 UTC m=+248.872049760" watchObservedRunningTime="2026-07-22 01:49:30.97256368 +0000 UTC m=+248.873442178" Jul 22 01:49:36.695244 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:36.695211 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:36.803939 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:36.803902 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:37.629628 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:37.629597 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:37.729744 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:37.729710 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:41.949944 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:41.949917 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-mptdp" Jul 22 01:49:44.262315 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:44.262227 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:45.492204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:45.492147 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:45.792952 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:45.792874 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:45.886243 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:45.886208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:46.019211 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:46.019172 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:46.095251 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:46.095154 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:46.971124 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:46.971092 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:50.435898 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.435859 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr"] Jul 22 01:49:50.439617 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.439592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.442282 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.442261 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 01:49:50.442426 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.442284 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 01:49:50.443409 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.443384 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 01:49:50.443528 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.443384 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:49:50.443592 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.443555 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 01:49:50.443592 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.443564 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5xqpc\"" Jul 22 01:49:50.448916 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.448890 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr"] Jul 22 01:49:50.497858 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.497815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.498031 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.497871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-metrics-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.498031 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.497937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80e8127a-9d88-4dc2-bf81-279ff889c680-manager-config\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.498031 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.498006 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567s2\" (UniqueName: \"kubernetes.io/projected/80e8127a-9d88-4dc2-bf81-279ff889c680-kube-api-access-567s2\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.598707 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.598660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.598900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.598803 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-metrics-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.598900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.598855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80e8127a-9d88-4dc2-bf81-279ff889c680-manager-config\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.599015 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.598914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567s2\" (UniqueName: \"kubernetes.io/projected/80e8127a-9d88-4dc2-bf81-279ff889c680-kube-api-access-567s2\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.599481 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.599458 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80e8127a-9d88-4dc2-bf81-279ff889c680-manager-config\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.601126 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.601096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.601239 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.601210 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80e8127a-9d88-4dc2-bf81-279ff889c680-metrics-cert\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.607264 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.607234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-567s2\" (UniqueName: \"kubernetes.io/projected/80e8127a-9d88-4dc2-bf81-279ff889c680-kube-api-access-567s2\") pod \"lws-controller-manager-7fb968c57-9dwjr\" (UID: \"80e8127a-9d88-4dc2-bf81-279ff889c680\") " pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.752248 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.752159 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:50.889651 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:50.889603 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr"] Jul 22 01:49:50.894910 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:49:50.894867 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e8127a_9d88_4dc2_bf81_279ff889c680.slice/crio-be0b93e10cb7d989fb30394213f0eeee5dfbc695beb578406fc95d2547bdf4db WatchSource:0}: Error finding container be0b93e10cb7d989fb30394213f0eeee5dfbc695beb578406fc95d2547bdf4db: Status 404 returned error can't find the container with id be0b93e10cb7d989fb30394213f0eeee5dfbc695beb578406fc95d2547bdf4db Jul 22 01:49:51.042585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:51.042496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" event={"ID":"80e8127a-9d88-4dc2-bf81-279ff889c680","Type":"ContainerStarted","Data":"be0b93e10cb7d989fb30394213f0eeee5dfbc695beb578406fc95d2547bdf4db"} Jul 22 01:49:51.547340 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:51.547251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:51.641407 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:51.641377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:52.342377 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.342348 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb"] Jul 22 01:49:52.345266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.345242 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.348883 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.348860 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 01:49:52.349521 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.349339 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 01:49:52.349521 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.349488 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 01:49:52.349521 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.349492 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-58zj5\"" Jul 22 01:49:52.370449 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.370415 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb"] Jul 22 01:49:52.418412 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418412 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wvv\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-kube-api-access-l9wvv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418858 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418705 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bce386-0a6b-4408-bba8-505bacc37420-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418858 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418764 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bce386-0a6b-4408-bba8-505bacc37420-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418858 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.418966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.418879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.427420 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.427389 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:52.519834 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.519793 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bce386-0a6b-4408-bba8-505bacc37420-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.519999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.519862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.519999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.519900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.519999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.519928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.519999 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.519962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520013 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wvv\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-kube-api-access-l9wvv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520043 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520231 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bce386-0a6b-4408-bba8-505bacc37420-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520433 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520400 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520433 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520532 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520472 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.520836 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.520791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.521149 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.521124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bce386-0a6b-4408-bba8-505bacc37420-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.522840 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.522814 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bce386-0a6b-4408-bba8-505bacc37420-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.522926 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.522858 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bce386-0a6b-4408-bba8-505bacc37420-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.530990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.530933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wvv\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-kube-api-access-l9wvv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.531845 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.531815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bce386-0a6b-4408-bba8-505bacc37420-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678d87zb\" (UID: \"85bce386-0a6b-4408-bba8-505bacc37420\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.659185 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.659097 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:52.689325 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.689276 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:52.763526 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.763489 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh"] Jul 22 01:49:52.766669 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.766629 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.769707 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.769681 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2xvbm\"" Jul 22 01:49:52.770098 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.770070 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 01:49:52.770198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.770058 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 01:49:52.770198 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.770073 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 01:49:52.779119 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.779095 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh"] Jul 22 01:49:52.784265 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.784234 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:52.925182 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrvm\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-kube-api-access-lrrvm\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925336 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925381 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925373 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04e3ce6-646c-4d27-b103-77ce49aed263-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925543 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925409 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:52.925543 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:52.925464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026382 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026572 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026572 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026516 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026572 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026767 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrvm\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-kube-api-access-lrrvm\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026767 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.026767 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.026707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04e3ce6-646c-4d27-b103-77ce49aed263-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.027304 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.027168 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.029257 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.029232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04e3ce6-646c-4d27-b103-77ce49aed263-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.029490 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.029460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.029490 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.029479 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.029894 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.029873 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.039369 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.039337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrvm\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-kube-api-access-lrrvm\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.039741 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.039528 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04e3ce6-646c-4d27-b103-77ce49aed263-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-lt6rh\" (UID: \"b04e3ce6-646c-4d27-b103-77ce49aed263\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.079293 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.079249 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:53.686900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.686834 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh"] Jul 22 01:49:53.688770 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:53.688743 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb"] Jul 22 01:49:53.690592 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:49:53.690572 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04e3ce6_646c_4d27_b103_77ce49aed263.slice/crio-13b83cac65cf4f0571a7aeebd380477aedec11e825a4719100db655b1cfd8d2c WatchSource:0}: Error finding container 13b83cac65cf4f0571a7aeebd380477aedec11e825a4719100db655b1cfd8d2c: Status 404 returned error can't find the container with id 13b83cac65cf4f0571a7aeebd380477aedec11e825a4719100db655b1cfd8d2c Jul 22 01:49:53.692954 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:49:53.692913 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bce386_0a6b_4408_bba8_505bacc37420.slice/crio-61a9dc7c2116e96daf51b8f9d913f45b60f67972548e4e1c365a796db5df3e32 WatchSource:0}: Error finding container 61a9dc7c2116e96daf51b8f9d913f45b60f67972548e4e1c365a796db5df3e32: Status 404 returned error can't find the container with id 61a9dc7c2116e96daf51b8f9d913f45b60f67972548e4e1c365a796db5df3e32 Jul 22 01:49:54.062011 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:54.061979 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" event={"ID":"b04e3ce6-646c-4d27-b103-77ce49aed263","Type":"ContainerStarted","Data":"13b83cac65cf4f0571a7aeebd380477aedec11e825a4719100db655b1cfd8d2c"} Jul 22 01:49:54.063174 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:54.063151 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" event={"ID":"85bce386-0a6b-4408-bba8-505bacc37420","Type":"ContainerStarted","Data":"61a9dc7c2116e96daf51b8f9d913f45b60f67972548e4e1c365a796db5df3e32"} Jul 22 01:49:54.064541 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:54.064512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" event={"ID":"80e8127a-9d88-4dc2-bf81-279ff889c680","Type":"ContainerStarted","Data":"238c92c40223bc42e2c6c9f5e59e25ec78778dbc1209e28ad20713455264efe6"} Jul 22 01:49:54.064678 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:54.064647 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:49:54.095747 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:54.095688 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" podStartSLOduration=1.444464955 podStartE2EDuration="4.095669414s" totalImagesPullingTime="2.651204459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:54.092801166 +0000 UTC m=+271.993679664" watchObservedRunningTime="2026-07-22 01:49:54.095669414 +0000 UTC m=+271.996547910" Jul 22 01:49:55.337219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:55.337171 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:55.429615 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:55.429578 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:55.760129 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:55.760089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:56.046014 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:56.045917 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:56.114129 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:56.114095 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:49:57.516717 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:57.516668 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:49:57.517108 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:57.516759 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:49:57.520717 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:57.520690 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:49:57.520807 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:57.520755 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:49:57.520807 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:57.520786 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:49:58.089230 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.089194 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" event={"ID":"b04e3ce6-646c-4d27-b103-77ce49aed263","Type":"ContainerStarted","Data":"f690fb61e8ddfd5c46c47196ac6ac19a68bd0212935f10ee9871eb5b8f875f5f"} Jul 22 01:49:58.089438 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.089364 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:58.090936 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.090905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" event={"ID":"85bce386-0a6b-4408-bba8-505bacc37420","Type":"ContainerStarted","Data":"68b3a53adc4857f426058cb9cd061a9e3f6e480f835c06bd1dd59230ca27c6eb"} Jul 22 01:49:58.091276 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.091251 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-lt6rh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 01:49:58.091349 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.091307 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" podUID="b04e3ce6-646c-4d27-b103-77ce49aed263" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 01:49:58.114125 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.114061 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" podStartSLOduration=2.290973658 podStartE2EDuration="6.114040882s" totalImagesPullingTime="3.823067224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:58.110477164 +0000 UTC m=+276.011355711" watchObservedRunningTime="2026-07-22 01:49:58.114040882 +0000 UTC m=+276.014919379" Jul 22 01:49:58.133883 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.133826 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" podStartSLOduration=2.308554453 podStartE2EDuration="6.133808949s" totalImagesPullingTime="3.825254496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:58.131712499 +0000 UTC m=+276.032590999" watchObservedRunningTime="2026-07-22 01:49:58.133808949 +0000 UTC m=+276.034687446" Jul 22 01:49:58.659930 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.659895 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:58.664973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:58.664945 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:59.095850 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:59.095813 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:49:59.096331 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:59.096310 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-lt6rh" Jul 22 01:49:59.096608 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:49:59.096592 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678d87zb" Jul 22 01:50:05.073901 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:05.073864 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7fb968c57-9dwjr" Jul 22 01:50:22.461445 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:22.461419 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 01:50:45.775766 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:45.775731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:50:45.878571 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:45.878538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:50:45.976622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:45.976580 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:50:46.719413 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:46.719379 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:50:46.931000 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:50:46.930967 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:14.059513 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:14.059420 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:21.062660 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:21.062604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:21.861962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:21.861924 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:26.684093 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:26.684060 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:29.613966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:29.613931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:30.188277 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.188243 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:30.267497 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.267459 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fdb66d479-pd6dc"] Jul 22 01:51:30.271155 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.271130 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.282290 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.282265 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fdb66d479-pd6dc"] Jul 22 01:51:30.337040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.337009 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:30.349327 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349292 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-trusted-ca-bundle\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349446 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349336 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndl2v\" (UniqueName: \"kubernetes.io/projected/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-kube-api-access-ndl2v\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349490 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349445 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-oauth-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349490 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349556 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349510 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-oauth-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349556 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-service-ca\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.349621 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.349572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.434258 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.434228 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:30.450563 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450473 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndl2v\" (UniqueName: \"kubernetes.io/projected/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-kube-api-access-ndl2v\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450563 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450528 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-oauth-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450563 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450562 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-oauth-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450604 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-service-ca\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.450882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.450784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-trusted-ca-bundle\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.451421 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.451397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.451531 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.451397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-oauth-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.451531 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.451454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-service-ca\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.451677 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.451559 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-trusted-ca-bundle\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.453278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.453252 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-serving-cert\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.453369 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.453357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-console-oauth-config\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.459933 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.459909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndl2v\" (UniqueName: \"kubernetes.io/projected/72cff911-4bc1-4802-9c4b-dfe6923f6fdf-kube-api-access-ndl2v\") pod \"console-fdb66d479-pd6dc\" (UID: \"72cff911-4bc1-4802-9c4b-dfe6923f6fdf\") " pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.580973 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.580917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:30.673061 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.672972 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:30.724883 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:30.724858 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fdb66d479-pd6dc"] Jul 22 01:51:30.727300 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:51:30.727272 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cff911_4bc1_4802_9c4b_dfe6923f6fdf.slice/crio-2611d92b428df5c95be37bb3f961382117d6c61d5aa26a20b071e6e698ab162e WatchSource:0}: Error finding container 2611d92b428df5c95be37bb3f961382117d6c61d5aa26a20b071e6e698ab162e: Status 404 returned error can't find the container with id 2611d92b428df5c95be37bb3f961382117d6c61d5aa26a20b071e6e698ab162e Jul 22 01:51:31.535880 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:31.535844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fdb66d479-pd6dc" event={"ID":"72cff911-4bc1-4802-9c4b-dfe6923f6fdf","Type":"ContainerStarted","Data":"a1891b59a916f16c72fe8e9c5b68888e80e0d5bc3f962cc8808807267dee0af4"} Jul 22 01:51:31.536093 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:31.535895 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fdb66d479-pd6dc" event={"ID":"72cff911-4bc1-4802-9c4b-dfe6923f6fdf","Type":"ContainerStarted","Data":"2611d92b428df5c95be37bb3f961382117d6c61d5aa26a20b071e6e698ab162e"} Jul 22 01:51:31.556556 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:31.556502 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fdb66d479-pd6dc" podStartSLOduration=1.5564836930000001 podStartE2EDuration="1.556483693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:31.555400591 +0000 UTC m=+369.456279118" watchObservedRunningTime="2026-07-22 01:51:31.556483693 +0000 UTC m=+369.457362189" Jul 22 01:51:34.637040 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:34.637005 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:40.082746 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.082711 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs"] Jul 22 01:51:40.086433 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.086408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.090135 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.090113 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l7cfb\"" Jul 22 01:51:40.090523 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.090506 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 01:51:40.090799 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.090783 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 01:51:40.109278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.109251 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs"] Jul 22 01:51:40.152504 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.152465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jf6\" (UniqueName: \"kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.152715 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.152661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.254074 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.254018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.254074 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.254086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jf6\" (UniqueName: \"kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.254529 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.254510 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.262939 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.262906 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jf6\" (UniqueName: \"kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6\") pod \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.397471 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.397377 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.525526 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.525493 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs"] Jul 22 01:51:40.535401 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.535371 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs"] Jul 22 01:51:40.553516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.553472 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:51:40.558514 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.558483 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.575877 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.575848 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:51:40.582204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.581859 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:40.582204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.581924 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:40.587822 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.587793 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:40.659967 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.659935 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.660119 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.660007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.663156 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:40.663123 2585 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 22 01:51:40.663156 ip-10-0-133-122 kubenswrapper[2585]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-n7mrs_kuadrant-system_942f9f95-025a-4c82-a656-37beb18e87c5_0(49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-n7mrs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4" Netns:"/var/run/netns/7faa38ce-6eaf-482b-9691-44b892d3aca4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-n7mrs;K8S_POD_INFRA_CONTAINER_ID=49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4;K8S_POD_UID=942f9f95-025a-4c82-a656-37beb18e87c5" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs/942f9f95-025a-4c82-a656-37beb18e87c5]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-n7mrs in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-n7mrs in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs: pods "kuadrant-operator-controller-manager-55467d9f7-n7mrs" not found Jul 22 01:51:40.663156 ip-10-0-133-122 kubenswrapper[2585]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 22 01:51:40.663156 ip-10-0-133-122 kubenswrapper[2585]: > Jul 22 01:51:40.663365 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:40.663193 2585 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 22 01:51:40.663365 ip-10-0-133-122 kubenswrapper[2585]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-n7mrs_kuadrant-system_942f9f95-025a-4c82-a656-37beb18e87c5_0(49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-n7mrs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4" Netns:"/var/run/netns/7faa38ce-6eaf-482b-9691-44b892d3aca4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-n7mrs;K8S_POD_INFRA_CONTAINER_ID=49677127ebbcd17d90274630f80253a1e61c6284abc535c080a8a2bc38ddc6a4;K8S_POD_UID=942f9f95-025a-4c82-a656-37beb18e87c5" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs/942f9f95-025a-4c82-a656-37beb18e87c5]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-n7mrs in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-n7mrs in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs: pods "kuadrant-operator-controller-manager-55467d9f7-n7mrs" not found Jul 22 01:51:40.663365 ip-10-0-133-122 kubenswrapper[2585]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 22 01:51:40.663365 ip-10-0-133-122 kubenswrapper[2585]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:40.761037 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.760982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.761037 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.761045 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.761439 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.761417 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.769696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.769664 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp\") pod \"kuadrant-operator-controller-manager-55467d9f7-7nqql\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:40.872219 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:40.872176 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:41.017159 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.017133 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:51:41.019554 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:51:41.019527 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a6b7db_af25_4c7b_90b0_c1d62a7d369e.slice/crio-82b310d1dca2db6bb93c2691dd416377f23ce481ead7f000641f0cc07008c408 WatchSource:0}: Error finding container 82b310d1dca2db6bb93c2691dd416377f23ce481ead7f000641f0cc07008c408: Status 404 returned error can't find the container with id 82b310d1dca2db6bb93c2691dd416377f23ce481ead7f000641f0cc07008c408 Jul 22 01:51:41.022024 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.022006 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:51:41.590534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.590499 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" event={"ID":"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e","Type":"ContainerStarted","Data":"82b310d1dca2db6bb93c2691dd416377f23ce481ead7f000641f0cc07008c408"} Jul 22 01:51:41.590971 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.590554 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:41.594428 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.594405 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fdb66d479-pd6dc" Jul 22 01:51:41.598152 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.598130 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:41.617823 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:41.617781 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" is forbidden: User \"system:node:ip-10-0-133-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-122.ec2.internal' and this object" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:41.656467 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.656431 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:51:41.659038 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:41.658990 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" is forbidden: User \"system:node:ip-10-0-133-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-122.ec2.internal' and this object" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:41.670520 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.670478 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2jf6\" (UniqueName: \"kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6\") pod \"942f9f95-025a-4c82-a656-37beb18e87c5\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " Jul 22 01:51:41.670718 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.670569 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume\") pod \"942f9f95-025a-4c82-a656-37beb18e87c5\" (UID: \"942f9f95-025a-4c82-a656-37beb18e87c5\") " Jul 22 01:51:41.671110 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.671062 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "942f9f95-025a-4c82-a656-37beb18e87c5" (UID: "942f9f95-025a-4c82-a656-37beb18e87c5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:51:41.673900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.673860 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6" (OuterVolumeSpecName: "kube-api-access-z2jf6") pod "942f9f95-025a-4c82-a656-37beb18e87c5" (UID: "942f9f95-025a-4c82-a656-37beb18e87c5"). InnerVolumeSpecName "kube-api-access-z2jf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:51:41.772706 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.772674 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jf6\" (UniqueName: \"kubernetes.io/projected/942f9f95-025a-4c82-a656-37beb18e87c5-kube-api-access-z2jf6\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:51:41.772706 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.772706 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/942f9f95-025a-4c82-a656-37beb18e87c5-extensions-socket-volume\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:51:41.776568 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.776532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:41.876307 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.876242 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:41.977071 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:41.977036 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:42.082698 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:42.082667 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:42.596425 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:42.596394 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:42.599066 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:42.599029 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" is forbidden: User \"system:node:ip-10-0-133-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-122.ec2.internal' and this object" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:42.606095 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:42.606064 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" is forbidden: User \"system:node:ip-10-0-133-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-122.ec2.internal' and this object" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:42.677204 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:51:42.677159 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-n7mrs\" is forbidden: User \"system:node:ip-10-0-133-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-122.ec2.internal' and this object" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-n7mrs" Jul 22 01:51:42.680494 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:42.680458 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942f9f95-025a-4c82-a656-37beb18e87c5" path="/var/lib/kubelet/pods/942f9f95-025a-4c82-a656-37beb18e87c5/volumes" Jul 22 01:51:44.910038 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:44.909992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:45.010249 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:45.010211 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:45.619589 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:45.619497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" event={"ID":"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e","Type":"ContainerStarted","Data":"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961"} Jul 22 01:51:45.619756 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:45.619622 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:45.641947 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:45.641895 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" podStartSLOduration=1.511099103 podStartE2EDuration="5.641876648s" totalImagesPullingTime="4.130777545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:45.639282998 +0000 UTC m=+383.540161514" watchObservedRunningTime="2026-07-22 01:51:45.641876648 +0000 UTC m=+383.542755144" Jul 22 01:51:46.066664 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:46.066589 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:46.125676 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:46.125621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:46.516027 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:46.515992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:46.676919 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:46.676872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:46.773169 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:46.773089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:47.921923 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:47.921884 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:49.707511 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:49.707477 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:50.725955 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:50.725889 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:50.824176 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:50.824146 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:54.668616 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:54.668576 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:54.769367 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:54.769332 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:56.627396 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:56.627357 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:51:58.477946 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:58.477908 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:51:58.582142 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:51:58.582102 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:06.681551 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:06.681511 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c4ccb47-d84cg" podUID="140f31c2-3023-431b-bee0-02b843e7c1a9" containerName="console" containerID="cri-o://d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845" gracePeriod=15 Jul 22 01:52:06.877066 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:06.877028 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:06.935584 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:06.935516 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:52:06.959988 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:06.959944 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:07.021337 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021307 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v4qm\" (UniqueName: \"kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021368 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021411 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021428 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021449 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021517 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.021597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021568 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert\") pod \"140f31c2-3023-431b-bee0-02b843e7c1a9\" (UID: \"140f31c2-3023-431b-bee0-02b843e7c1a9\") " Jul 22 01:52:07.022000 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.021936 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:52:07.022202 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.022161 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:52:07.022202 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.022145 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config" (OuterVolumeSpecName: "console-config") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:52:07.023732 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.022396 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:52:07.024347 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.024274 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm" (OuterVolumeSpecName: "kube-api-access-5v4qm") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "kube-api-access-5v4qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:52:07.024474 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.024354 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:52:07.024559 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.024529 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "140f31c2-3023-431b-bee0-02b843e7c1a9" (UID: "140f31c2-3023-431b-bee0-02b843e7c1a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:52:07.123002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.122960 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.122995 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.123005 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123002 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.123015 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.123023 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/140f31c2-3023-431b-bee0-02b843e7c1a9-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.123031 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/140f31c2-3023-431b-bee0-02b843e7c1a9-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.123278 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.123039 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v4qm\" (UniqueName: \"kubernetes.io/projected/140f31c2-3023-431b-bee0-02b843e7c1a9-kube-api-access-5v4qm\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:07.728247 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.728212 2585 generic.go:356] "Generic (PLEG): container finished" podID="140f31c2-3023-431b-bee0-02b843e7c1a9" containerID="d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845" exitCode=0 Jul 22 01:52:07.728247 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.728242 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c4ccb47-d84cg" event={"ID":"140f31c2-3023-431b-bee0-02b843e7c1a9","Type":"ContainerDied","Data":"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845"} Jul 22 01:52:07.728739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.728257 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c4ccb47-d84cg" Jul 22 01:52:07.728739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.728296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c4ccb47-d84cg" event={"ID":"140f31c2-3023-431b-bee0-02b843e7c1a9","Type":"ContainerDied","Data":"9b414f4c405d840a72cab5dfefe0a51566d095a66e282867a5f4e7f745817e01"} Jul 22 01:52:07.728739 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.728317 2585 scope.go:117] "RemoveContainer" containerID="d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845" Jul 22 01:52:07.738300 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.738281 2585 scope.go:117] "RemoveContainer" containerID="d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845" Jul 22 01:52:07.738563 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:52:07.738543 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845\": container with ID starting with d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845 not found: ID does not exist" containerID="d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845" Jul 22 01:52:07.738627 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.738574 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845"} err="failed to get container status \"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845\": rpc error: code = NotFound desc = could not find container \"d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845\": container with ID starting with d24bd1ae16046483ced469d85b7b65532018191c46f8cfdd073e65cfc5e6a845 not found: ID does not exist" Jul 22 01:52:07.749407 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.749371 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:52:07.753339 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:07.753313 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c4ccb47-d84cg"] Jul 22 01:52:08.677296 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:08.677260 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140f31c2-3023-431b-bee0-02b843e7c1a9" path="/var/lib/kubelet/pods/140f31c2-3023-431b-bee0-02b843e7c1a9/volumes" Jul 22 01:52:08.870171 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:08.870134 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:52:08.870561 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:08.870354 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" podUID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" containerName="manager" containerID="cri-o://2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961" gracePeriod=10 Jul 22 01:52:09.133491 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.133464 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:52:09.241440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.241405 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume\") pod \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " Jul 22 01:52:09.241612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.241469 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp\") pod \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\" (UID: \"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e\") " Jul 22 01:52:09.241959 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.241924 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" (UID: "b0a6b7db-af25-4c7b-90b0-c1d62a7d369e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:52:09.243620 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.243570 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp" (OuterVolumeSpecName: "kube-api-access-cpvjp") pod "b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" (UID: "b0a6b7db-af25-4c7b-90b0-c1d62a7d369e"). InnerVolumeSpecName "kube-api-access-cpvjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:52:09.342624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.342591 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-extensions-socket-volume\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:09.342624 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.342620 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvjp\" (UniqueName: \"kubernetes.io/projected/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e-kube-api-access-cpvjp\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:52:09.393206 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.393168 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:09.515551 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.515464 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:09.739478 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.739435 2585 generic.go:356] "Generic (PLEG): container finished" podID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" containerID="2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961" exitCode=0 Jul 22 01:52:09.739673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.739498 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" Jul 22 01:52:09.739673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.739507 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" event={"ID":"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e","Type":"ContainerDied","Data":"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961"} Jul 22 01:52:09.739673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.739555 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql" event={"ID":"b0a6b7db-af25-4c7b-90b0-c1d62a7d369e","Type":"ContainerDied","Data":"82b310d1dca2db6bb93c2691dd416377f23ce481ead7f000641f0cc07008c408"} Jul 22 01:52:09.739673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.739581 2585 scope.go:117] "RemoveContainer" containerID="2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961" Jul 22 01:52:09.748887 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.748867 2585 scope.go:117] "RemoveContainer" containerID="2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961" Jul 22 01:52:09.749174 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:52:09.749153 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961\": container with ID starting with 2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961 not found: ID does not exist" containerID="2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961" Jul 22 01:52:09.749241 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.749185 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961"} err="failed to get container status \"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961\": rpc error: code = NotFound desc = could not find container \"2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961\": container with ID starting with 2813ebe3f56e5f43de8a7ea13cd350a03d8b643ba59488674c823cd4fcaff961 not found: ID does not exist" Jul 22 01:52:09.762366 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.762333 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:52:09.765183 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:09.765153 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7nqql"] Jul 22 01:52:10.677452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:10.677416 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" path="/var/lib/kubelet/pods/b0a6b7db-af25-4c7b-90b0-c1d62a7d369e/volumes" Jul 22 01:52:13.088112 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088079 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6"] Jul 22 01:52:13.088699 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088668 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140f31c2-3023-431b-bee0-02b843e7c1a9" containerName="console" Jul 22 01:52:13.088699 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088694 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="140f31c2-3023-431b-bee0-02b843e7c1a9" containerName="console" Jul 22 01:52:13.088807 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088739 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" containerName="manager" Jul 22 01:52:13.088807 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088748 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" containerName="manager" Jul 22 01:52:13.088876 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088829 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a6b7db-af25-4c7b-90b0-c1d62a7d369e" containerName="manager" Jul 22 01:52:13.088876 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.088841 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="140f31c2-3023-431b-bee0-02b843e7c1a9" containerName="console" Jul 22 01:52:13.095661 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.095616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.098204 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.098177 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kj2c2\"" Jul 22 01:52:13.105251 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.105219 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6"] Jul 22 01:52:13.121181 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.121150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:13.180667 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8c0946-edc7-4881-a180-233176c150c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.180823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8c0946-edc7-4881-a180-233176c150c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.180823 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.180968 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.180968 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.180968 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.181106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.181106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.180995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.181106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.181046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp6x\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-kube-api-access-4fp6x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282597 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282553 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8c0946-edc7-4881-a180-233176c150c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8c0946-edc7-4881-a180-233176c150c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.282827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283193 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.282840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp6x\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-kube-api-access-4fp6x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283353 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.283332 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.283371 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283452 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.283443 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283561 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.283460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.283561 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.283551 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8c0946-edc7-4881-a180-233176c150c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.285133 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.285110 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8c0946-edc7-4881-a180-233176c150c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.285236 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.285158 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8c0946-edc7-4881-a180-233176c150c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.293655 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.293584 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.296873 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.296849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp6x\" (UniqueName: \"kubernetes.io/projected/be8c0946-edc7-4881-a180-233176c150c4-kube-api-access-4fp6x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-bkzb6\" (UID: \"be8c0946-edc7-4881-a180-233176c150c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.327685 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.327623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:13.408009 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.407915 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:13.553438 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.553414 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6"] Jul 22 01:52:13.555691 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:52:13.555662 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8c0946_edc7_4881_a180_233176c150c4.slice/crio-1c673d45fc46417decd7ea8b82d61afc245b41bdf57805cb64059c98e9cfac00 WatchSource:0}: Error finding container 1c673d45fc46417decd7ea8b82d61afc245b41bdf57805cb64059c98e9cfac00: Status 404 returned error can't find the container with id 1c673d45fc46417decd7ea8b82d61afc245b41bdf57805cb64059c98e9cfac00 Jul 22 01:52:13.557921 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.557889 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:52:13.558001 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.557956 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:52:13.558001 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.557991 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:52:13.762080 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.762042 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" event={"ID":"be8c0946-edc7-4881-a180-233176c150c4","Type":"ContainerStarted","Data":"4b4d11352217aea82afc3f704b8a7abbf999b9147413c1f3c8c580d326cbbea3"} Jul 22 01:52:13.762294 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.762095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" event={"ID":"be8c0946-edc7-4881-a180-233176c150c4","Type":"ContainerStarted","Data":"1c673d45fc46417decd7ea8b82d61afc245b41bdf57805cb64059c98e9cfac00"} Jul 22 01:52:13.784218 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:13.784164 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" podStartSLOduration=0.78414696 podStartE2EDuration="784.14696ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:52:13.783672492 +0000 UTC m=+411.684550988" watchObservedRunningTime="2026-07-22 01:52:13.78414696 +0000 UTC m=+411.685025458" Jul 22 01:52:14.358408 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.358377 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:14.408889 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.408853 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:14.413909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.413883 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:14.417539 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.417502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:14.767295 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.767266 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:14.768195 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:14.768174 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:52:18.084026 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:18.083991 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:18.142469 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:18.142440 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:28.732188 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:28.732149 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:28.831441 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:28.831408 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:29.429304 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:29.429269 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:29.550565 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:29.550524 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:29.651696 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:29.651664 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:30.081687 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:30.081623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:30.179482 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:30.179449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:30.297283 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:30.297251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:30.374015 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:30.373940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:34.472702 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:34.472665 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:34.528419 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:34.528387 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:38.877087 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:38.877043 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:38.977430 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:38.977389 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:51.686100 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:51.686022 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:51.783950 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:51.783916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:52:52.203774 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:52:52.203740 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:04.474373 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:04.474331 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:04.672882 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:04.672834 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:09.759625 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:09.759592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:11.148528 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.148490 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:11.165008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.164968 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:11.168622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.168606 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.171429 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.171402 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f48b9\"" Jul 22 01:53:11.171429 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.171403 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:53:11.171429 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.171428 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:53:11.178848 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.178822 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:11.247075 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.247044 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:11.273328 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.273298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.273465 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.273360 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nsg\" (UniqueName: \"kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.300146 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.300115 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:11.374598 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.374565 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.374835 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.374692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nsg\" (UniqueName: \"kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.377103 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.377075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.384411 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.384388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nsg\" (UniqueName: \"kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg\") pod \"maas-controller-79d7d5b874-2ztfx\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.410874 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.410791 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:11.480405 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.480370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:11.622735 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:11.622563 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:11.625696 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:53:11.625663 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142fff85_8e1d_4535_bc3e_801071c25db4.slice/crio-8c54ea30a2c47a1eb1c94c84ee4c8770ffc66e4104f1fc4a425a729a0d44d0cd WatchSource:0}: Error finding container 8c54ea30a2c47a1eb1c94c84ee4c8770ffc66e4104f1fc4a425a729a0d44d0cd: Status 404 returned error can't find the container with id 8c54ea30a2c47a1eb1c94c84ee4c8770ffc66e4104f1fc4a425a729a0d44d0cd Jul 22 01:53:12.048263 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:12.048223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" event={"ID":"142fff85-8e1d-4535-bc3e-801071c25db4","Type":"ContainerStarted","Data":"8c54ea30a2c47a1eb1c94c84ee4c8770ffc66e4104f1fc4a425a729a0d44d0cd"} Jul 22 01:53:14.232421 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:14.232387 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:14.349912 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:14.349881 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:15.067530 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:15.067491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" event={"ID":"142fff85-8e1d-4535-bc3e-801071c25db4","Type":"ContainerStarted","Data":"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba"} Jul 22 01:53:15.067758 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:15.067614 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:15.102757 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:15.102695 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" podStartSLOduration=1.476858999 podStartE2EDuration="4.102676462s" totalImagesPullingTime="2.625817463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:15.100459267 +0000 UTC m=+473.001337777" watchObservedRunningTime="2026-07-22 01:53:15.102676462 +0000 UTC m=+473.003554958" Jul 22 01:53:25.217969 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.217933 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:25.218420 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.218131 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" podUID="142fff85-8e1d-4535-bc3e-801071c25db4" containerName="manager" containerID="cri-o://9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba" gracePeriod=30 Jul 22 01:53:25.220068 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.220042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:25.468453 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.468383 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:25.617088 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.617052 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert\") pod \"142fff85-8e1d-4535-bc3e-801071c25db4\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " Jul 22 01:53:25.617270 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.617125 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24nsg\" (UniqueName: \"kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg\") pod \"142fff85-8e1d-4535-bc3e-801071c25db4\" (UID: \"142fff85-8e1d-4535-bc3e-801071c25db4\") " Jul 22 01:53:25.619357 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.619320 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "142fff85-8e1d-4535-bc3e-801071c25db4" (UID: "142fff85-8e1d-4535-bc3e-801071c25db4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:53:25.619506 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.619388 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg" (OuterVolumeSpecName: "kube-api-access-24nsg") pod "142fff85-8e1d-4535-bc3e-801071c25db4" (UID: "142fff85-8e1d-4535-bc3e-801071c25db4"). InnerVolumeSpecName "kube-api-access-24nsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:53:25.718793 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.718701 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nsg\" (UniqueName: \"kubernetes.io/projected/142fff85-8e1d-4535-bc3e-801071c25db4-kube-api-access-24nsg\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:53:25.718793 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:25.718736 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/142fff85-8e1d-4535-bc3e-801071c25db4-webhook-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:53:26.122034 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.121931 2585 generic.go:356] "Generic (PLEG): container finished" podID="142fff85-8e1d-4535-bc3e-801071c25db4" containerID="9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba" exitCode=0 Jul 22 01:53:26.122034 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.122000 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" Jul 22 01:53:26.122220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.122027 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" event={"ID":"142fff85-8e1d-4535-bc3e-801071c25db4","Type":"ContainerDied","Data":"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba"} Jul 22 01:53:26.122220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.122087 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79d7d5b874-2ztfx" event={"ID":"142fff85-8e1d-4535-bc3e-801071c25db4","Type":"ContainerDied","Data":"8c54ea30a2c47a1eb1c94c84ee4c8770ffc66e4104f1fc4a425a729a0d44d0cd"} Jul 22 01:53:26.122220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.122109 2585 scope.go:117] "RemoveContainer" containerID="9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba" Jul 22 01:53:26.131609 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.131587 2585 scope.go:117] "RemoveContainer" containerID="9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba" Jul 22 01:53:26.131959 ip-10-0-133-122 kubenswrapper[2585]: E0722 01:53:26.131936 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba\": container with ID starting with 9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba not found: ID does not exist" containerID="9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba" Jul 22 01:53:26.132031 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.131968 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba"} err="failed to get container status \"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba\": rpc error: code = NotFound desc = could not find container \"9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba\": container with ID starting with 9083a07a56f150ff51c204cd04dc184029d7350130c99854a82ba3ce578b51ba not found: ID does not exist" Jul 22 01:53:26.144250 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.144222 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:26.147924 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.147899 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79d7d5b874-2ztfx"] Jul 22 01:53:26.677585 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:26.677546 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="142fff85-8e1d-4535-bc3e-801071c25db4" path="/var/lib/kubelet/pods/142fff85-8e1d-4535-bc3e-801071c25db4/volumes" Jul 22 01:53:46.004631 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.004573 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:46.100294 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.100260 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:46.718407 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.718360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:46.785106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.785069 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5"] Jul 22 01:53:46.785535 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.785504 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142fff85-8e1d-4535-bc3e-801071c25db4" containerName="manager" Jul 22 01:53:46.785535 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.785528 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="142fff85-8e1d-4535-bc3e-801071c25db4" containerName="manager" Jul 22 01:53:46.785787 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.785681 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="142fff85-8e1d-4535-bc3e-801071c25db4" containerName="manager" Jul 22 01:53:46.787689 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.787671 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:46.790516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.790490 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 01:53:46.791614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.791598 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 01:53:46.791614 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.791608 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 01:53:46.791828 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.791617 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\"" Jul 22 01:53:46.798647 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.798609 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5"] Jul 22 01:53:46.909550 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.909497 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:46.925889 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.925852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrkl\" (UniqueName: \"kubernetes.io/projected/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-kube-api-access-mfrkl\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:46.926066 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.925959 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-maas-api-tls\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:46.926066 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:46.926055 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-openshift-service-ca\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.027419 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.027335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-maas-api-tls\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.027419 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.027399 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-openshift-service-ca\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.027958 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.027444 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrkl\" (UniqueName: \"kubernetes.io/projected/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-kube-api-access-mfrkl\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.028211 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.028188 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-openshift-service-ca\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.029891 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.029872 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-maas-api-tls\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.038684 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.038662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrkl\" (UniqueName: \"kubernetes.io/projected/e5685683-c0f3-44fc-b99c-a3a026ef5e2f-kube-api-access-mfrkl\") pod \"maas-api-685bd66c77-vhnb5\" (UID: \"e5685683-c0f3-44fc-b99c-a3a026ef5e2f\") " pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.098940 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.098905 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:47.272654 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.272609 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5"] Jul 22 01:53:47.277011 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:53:47.276975 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5685683_c0f3_44fc_b99c_a3a026ef5e2f.slice/crio-36f892ee0dd7a2aec15cccaa4b23f1571c7eec69477036ef0993c0075c5b0f02 WatchSource:0}: Error finding container 36f892ee0dd7a2aec15cccaa4b23f1571c7eec69477036ef0993c0075c5b0f02: Status 404 returned error can't find the container with id 36f892ee0dd7a2aec15cccaa4b23f1571c7eec69477036ef0993c0075c5b0f02 Jul 22 01:53:47.316416 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.316383 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:47.418749 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.418707 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:47.464732 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.464694 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9mfhx"] Jul 22 01:53:47.468356 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.468334 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.471760 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.471741 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 01:53:47.471872 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.471759 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hhprp\"" Jul 22 01:53:47.478701 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.478677 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9mfhx"] Jul 22 01:53:47.532540 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.532434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8kn\" (UniqueName: \"kubernetes.io/projected/c1790ce0-5b8d-487a-9859-b6054abaf5ba-kube-api-access-gd8kn\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.532540 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.532520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1790ce0-5b8d-487a-9859-b6054abaf5ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.633956 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.633922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8kn\" (UniqueName: \"kubernetes.io/projected/c1790ce0-5b8d-487a-9859-b6054abaf5ba-kube-api-access-gd8kn\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.634126 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.633999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1790ce0-5b8d-487a-9859-b6054abaf5ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.635020 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.634994 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1790ce0-5b8d-487a-9859-b6054abaf5ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.642562 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.642538 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8kn\" (UniqueName: \"kubernetes.io/projected/c1790ce0-5b8d-487a-9859-b6054abaf5ba-kube-api-access-gd8kn\") pod \"payload-processing-5bf47cd988-9mfhx\" (UID: \"c1790ce0-5b8d-487a-9859-b6054abaf5ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.781990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.781952 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:47.915849 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:47.915815 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9mfhx"] Jul 22 01:53:47.918604 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:53:47.918571 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1790ce0_5b8d_487a_9859_b6054abaf5ba.slice/crio-c1471c28fd2c1403718c3ba61300ca274a4c5bcbf8a6ed38780741878d941837 WatchSource:0}: Error finding container c1471c28fd2c1403718c3ba61300ca274a4c5bcbf8a6ed38780741878d941837: Status 404 returned error can't find the container with id c1471c28fd2c1403718c3ba61300ca274a4c5bcbf8a6ed38780741878d941837 Jul 22 01:53:48.248821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:48.248768 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" event={"ID":"c1790ce0-5b8d-487a-9859-b6054abaf5ba","Type":"ContainerStarted","Data":"c1471c28fd2c1403718c3ba61300ca274a4c5bcbf8a6ed38780741878d941837"} Jul 22 01:53:48.250541 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:48.250475 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" event={"ID":"e5685683-c0f3-44fc-b99c-a3a026ef5e2f","Type":"ContainerStarted","Data":"36f892ee0dd7a2aec15cccaa4b23f1571c7eec69477036ef0993c0075c5b0f02"} Jul 22 01:53:49.258679 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:49.258617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" event={"ID":"e5685683-c0f3-44fc-b99c-a3a026ef5e2f","Type":"ContainerStarted","Data":"35d38a99a40d29900e5f4d30f9b936bb84298d134ff2831bd7874037e8b95623"} Jul 22 01:53:49.259150 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:49.258701 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:49.277106 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:49.277000 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" podStartSLOduration=1.54278311 podStartE2EDuration="3.276978177s" totalImagesPullingTime="1.734195067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:49.274441918 +0000 UTC m=+507.175320415" watchObservedRunningTime="2026-07-22 01:53:49.276978177 +0000 UTC m=+507.177856675" Jul 22 01:53:50.265359 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:50.265321 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" event={"ID":"c1790ce0-5b8d-487a-9859-b6054abaf5ba","Type":"ContainerStarted","Data":"0487412a8457e24c32923ed82882358c62ab0e9a82daa5f3b7a04ad89f2146e2"} Jul 22 01:53:50.282479 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:50.282426 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" podStartSLOduration=1.344236336 podStartE2EDuration="3.282408063s" totalImagesPullingTime="1.938171727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:50.280592056 +0000 UTC m=+508.181470590" watchObservedRunningTime="2026-07-22 01:53:50.282408063 +0000 UTC m=+508.183286560" Jul 22 01:53:51.269569 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:51.269528 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:53:55.271078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.271004 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-685bd66c77-vhnb5" Jul 22 01:53:55.293271 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.293189 2585 generic.go:356] "Generic (PLEG): container finished" podID="be8c0946-edc7-4881-a180-233176c150c4" containerID="4b4d11352217aea82afc3f704b8a7abbf999b9147413c1f3c8c580d326cbbea3" exitCode=0 Jul 22 01:53:55.293271 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.293231 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" event={"ID":"be8c0946-edc7-4881-a180-233176c150c4","Type":"ContainerDied","Data":"4b4d11352217aea82afc3f704b8a7abbf999b9147413c1f3c8c580d326cbbea3"} Jul 22 01:53:55.293673 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.293660 2585 scope.go:117] "RemoveContainer" containerID="4b4d11352217aea82afc3f704b8a7abbf999b9147413c1f3c8c580d326cbbea3" Jul 22 01:53:55.294809 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.294778 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:53:55.294914 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.294857 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:53:55.294956 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:55.294914 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 01:53:56.306628 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:56.306589 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" event={"ID":"be8c0946-edc7-4881-a180-233176c150c4","Type":"ContainerStarted","Data":"239e85af9150993d66d371657ad0c453291b4861a938fbe8f48b9a97ac79be0f"} Jul 22 01:53:56.408827 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:56.408785 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:53:56.409826 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:56.409801 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bkzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:53:56.409950 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:56.409860 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" podUID="be8c0946-edc7-4881-a180-233176c150c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:53:57.069078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:57.069027 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:57.171118 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:57.171082 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:53:57.408986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:57.408900 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bkzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:53:57.408986 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:57.408958 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" podUID="be8c0946-edc7-4881-a180-233176c150c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:53:58.408968 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:58.408928 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bkzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:53:58.409157 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:58.408987 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" podUID="be8c0946-edc7-4881-a180-233176c150c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:53:59.409024 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:59.408986 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-bkzb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:53:59.409220 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:59.409046 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" podUID="be8c0946-edc7-4881-a180-233176c150c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:53:59.767737 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:53:59.767703 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:54:00.412490 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:00.412461 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:54:00.413415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:00.413396 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-bkzb6" Jul 22 01:54:02.277631 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:02.277601 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-9mfhx" Jul 22 01:54:03.127857 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:03.127820 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:09.395480 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.395446 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s"] Jul 22 01:54:09.398367 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.398350 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.401127 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.401108 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 01:54:09.402167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.402151 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8bf95\"" Jul 22 01:54:09.402230 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.402195 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 01:54:09.402280 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.402195 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 01:54:09.408666 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.408609 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s"] Jul 22 01:54:09.476466 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.476466 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476468 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.476708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.476708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62vgk\" (UniqueName: \"kubernetes.io/projected/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-kube-api-access-62vgk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.476708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476611 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.476708 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.476662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.502431 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.502384 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:09.577205 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577169 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577224 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vgk\" (UniqueName: \"kubernetes.io/projected/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-kube-api-access-62vgk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577454 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577820 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577820 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577791 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.577921 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.577828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.579622 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.579606 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.579900 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.579884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.585902 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.585881 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vgk\" (UniqueName: \"kubernetes.io/projected/60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131-kube-api-access-62vgk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-87k4s\" (UID: \"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.709335 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.709295 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:09.842395 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:09.842371 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s"] Jul 22 01:54:09.844198 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:54:09.844171 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ffb6dd_f3ed_4a17_87e3_a8f2f7ed2131.slice/crio-c36e5bd6d731935e5d9e0beace5dedaa50de3e1552b5d92d66c415062fb0f2e2 WatchSource:0}: Error finding container c36e5bd6d731935e5d9e0beace5dedaa50de3e1552b5d92d66c415062fb0f2e2: Status 404 returned error can't find the container with id c36e5bd6d731935e5d9e0beace5dedaa50de3e1552b5d92d66c415062fb0f2e2 Jul 22 01:54:10.380958 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:10.380915 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" event={"ID":"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131","Type":"ContainerStarted","Data":"c36e5bd6d731935e5d9e0beace5dedaa50de3e1552b5d92d66c415062fb0f2e2"} Jul 22 01:54:11.387167 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:11.387131 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" event={"ID":"60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131","Type":"ContainerStarted","Data":"4419bc8a9e95927ecabfd743d73f66952c1d7efbfa825fe9a4e63c95c63d524c"} Jul 22 01:54:11.406066 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:11.406015 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" podStartSLOduration=1.424099838 podStartE2EDuration="2.405996346s" totalImagesPullingTime="981.896508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:11.405056666 +0000 UTC m=+529.305935164" watchObservedRunningTime="2026-07-22 01:54:11.405996346 +0000 UTC m=+529.306874842" Jul 22 01:54:19.710025 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:19.709985 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:19.710539 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:19.710058 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:19.722677 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:19.722625 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:20.443918 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:20.443888 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-87k4s" Jul 22 01:54:22.487711 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.487682 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk"] Jul 22 01:54:22.490894 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.490871 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.493747 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.493725 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 01:54:22.501662 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.501610 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk"] Jul 22 01:54:22.516875 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.516838 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:22.569021 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.568984 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:22.612709 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.612909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612724 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.612909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612763 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnc6\" (UniqueName: \"kubernetes.io/projected/a585b287-a7ae-4ceb-8435-5d9a72868c38-kube-api-access-4vnc6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.612909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a585b287-a7ae-4ceb-8435-5d9a72868c38-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.613041 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.613041 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.612957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714348 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714549 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714356 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnc6\" (UniqueName: \"kubernetes.io/projected/a585b287-a7ae-4ceb-8435-5d9a72868c38-kube-api-access-4vnc6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714549 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a585b287-a7ae-4ceb-8435-5d9a72868c38-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714549 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714549 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714464 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714549 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714872 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714948 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714920 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.714985 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.714934 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.716591 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.716573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a585b287-a7ae-4ceb-8435-5d9a72868c38-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.716962 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.716943 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a585b287-a7ae-4ceb-8435-5d9a72868c38-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.722966 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.722946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnc6\" (UniqueName: \"kubernetes.io/projected/a585b287-a7ae-4ceb-8435-5d9a72868c38-kube-api-access-4vnc6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gbdkk\" (UID: \"a585b287-a7ae-4ceb-8435-5d9a72868c38\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.802524 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.802417 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:22.940153 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:22.940128 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk"] Jul 22 01:54:22.942292 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:54:22.942267 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda585b287_a7ae_4ceb_8435_5d9a72868c38.slice/crio-84428ebc06073c4241df2edaa2beac3d7285f8ab4c90598ab94525449ab5370a WatchSource:0}: Error finding container 84428ebc06073c4241df2edaa2beac3d7285f8ab4c90598ab94525449ab5370a: Status 404 returned error can't find the container with id 84428ebc06073c4241df2edaa2beac3d7285f8ab4c90598ab94525449ab5370a Jul 22 01:54:23.447449 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:23.447408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" event={"ID":"a585b287-a7ae-4ceb-8435-5d9a72868c38","Type":"ContainerStarted","Data":"d5ffc8a8abc281e860b7d8196acf1829a0aa2668af2695b4c19bcf36486b104c"} Jul 22 01:54:23.447670 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:23.447473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" event={"ID":"a585b287-a7ae-4ceb-8435-5d9a72868c38","Type":"ContainerStarted","Data":"84428ebc06073c4241df2edaa2beac3d7285f8ab4c90598ab94525449ab5370a"} Jul 22 01:54:23.466744 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:23.466629 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" podStartSLOduration=1.205726356 podStartE2EDuration="1.466611415s" totalImagesPullingTime="260.885059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:23.46456971 +0000 UTC m=+541.365448243" watchObservedRunningTime="2026-07-22 01:54:23.466611415 +0000 UTC m=+541.367489912" Jul 22 01:54:28.427709 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:28.427676 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:32.803196 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:32.803160 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:32.803606 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:32.803501 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:32.815733 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:32.815707 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:33.510368 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:33.510339 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gbdkk" Jul 22 01:54:35.507821 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:35.507785 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:54:42.416617 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:54:42.416581 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:21.110580 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:21.110546 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:22.493516 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:22.493486 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:48.182275 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:48.182189 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:48.275212 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:48.275178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:49.472266 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:49.472230 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:55:49.570668 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:55:49.570609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:56:01.232349 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:56:01.232305 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:56:01.289069 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:56:01.289037 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:00.646747 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:00.646710 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:00.713246 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:00.713213 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:04.016415 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:04.016376 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:38.248918 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.248877 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:38.251288 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.251271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.253935 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.253909 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:57:38.254086 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.253950 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f48b9\"" Jul 22 01:57:38.254981 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.254967 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:57:38.261612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.261591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:38.381720 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.381686 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsqc\" (UniqueName: \"kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.381914 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.381744 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.482551 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.482519 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsqc\" (UniqueName: \"kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.482737 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.482574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.485105 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.485083 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.493085 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.493058 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsqc\" (UniqueName: \"kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc\") pod \"maas-controller-969c84445-2t96z\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.562085 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.561997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:38.905377 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.905350 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:38.907001 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:57:38.906971 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b0fe99_0bc6_4ff6_8fd7_ea20873311b9.slice/crio-d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3 WatchSource:0}: Error finding container d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3: Status 404 returned error can't find the container with id d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3 Jul 22 01:57:38.908335 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:38.908318 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:57:39.411359 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:39.411328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-2t96z" event={"ID":"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9","Type":"ContainerStarted","Data":"5b0585bc3df11cf90cf4277faba607b9edb4caf9eff3c6f7bf8e45c2fcbb1b31"} Jul 22 01:57:39.411744 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:39.411406 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-2t96z" event={"ID":"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9","Type":"ContainerStarted","Data":"d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3"} Jul 22 01:57:39.411744 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:39.411426 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:39.430336 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:39.430286 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-969c84445-2t96z" podStartSLOduration=1.011489323 podStartE2EDuration="1.430264099s" totalImagesPullingTime="418.774776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:39.427224228 +0000 UTC m=+737.328102727" watchObservedRunningTime="2026-07-22 01:57:39.430264099 +0000 UTC m=+737.331142598" Jul 22 01:57:50.419990 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:50.419956 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:55.154765 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.154714 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:55.155233 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.154939 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-969c84445-2t96z" podUID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" containerName="manager" containerID="cri-o://5b0585bc3df11cf90cf4277faba607b9edb4caf9eff3c6f7bf8e45c2fcbb1b31" gracePeriod=30 Jul 22 01:57:55.159008 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.158982 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:55.492390 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.492362 2585 generic.go:356] "Generic (PLEG): container finished" podID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" containerID="5b0585bc3df11cf90cf4277faba607b9edb4caf9eff3c6f7bf8e45c2fcbb1b31" exitCode=0 Jul 22 01:57:55.492534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.492432 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-2t96z" event={"ID":"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9","Type":"ContainerDied","Data":"5b0585bc3df11cf90cf4277faba607b9edb4caf9eff3c6f7bf8e45c2fcbb1b31"} Jul 22 01:57:55.492534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.492500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-2t96z" event={"ID":"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9","Type":"ContainerDied","Data":"d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3"} Jul 22 01:57:55.492534 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.492526 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d094c8dd2702299e3fa5f67fbcfb636e4b2a0b840780109dbe550d07d3669fe3" Jul 22 01:57:55.504289 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.504267 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:55.538354 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.538327 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert\") pod \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " Jul 22 01:57:55.538536 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.538424 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klsqc\" (UniqueName: \"kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc\") pod \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\" (UID: \"b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9\") " Jul 22 01:57:55.540440 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.540406 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" (UID: "b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:57:55.540552 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.540509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc" (OuterVolumeSpecName: "kube-api-access-klsqc") pod "b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" (UID: "b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9"). InnerVolumeSpecName "kube-api-access-klsqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:57:55.640291 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.640253 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsqc\" (UniqueName: \"kubernetes.io/projected/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-kube-api-access-klsqc\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:57:55.640291 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:55.640285 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9-webhook-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 01:57:56.496909 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:56.496870 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-2t96z" Jul 22 01:57:56.518018 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:56.517985 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:56.522228 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:56.522199 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-969c84445-2t96z"] Jul 22 01:57:56.677331 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:56.677293 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" path="/var/lib/kubelet/pods/b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9/volumes" Jul 22 01:57:57.247697 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.247660 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:57:57.407209 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.407176 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-969c84445-tjqh6"] Jul 22 01:57:57.407609 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.407597 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" containerName="manager" Jul 22 01:57:57.407686 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.407611 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" containerName="manager" Jul 22 01:57:57.407723 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.407694 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b0fe99-0bc6-4ff6-8fd7-ea20873311b9" containerName="manager" Jul 22 01:57:57.409652 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.409621 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.413338 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.413314 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f48b9\"" Jul 22 01:57:57.413467 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.413315 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:57:57.413467 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.413323 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:57:57.419492 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.419464 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-tjqh6"] Jul 22 01:57:57.457420 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.457384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xsk\" (UniqueName: \"kubernetes.io/projected/39ab2936-c145-4f75-9549-1e4d47a9d5f6-kube-api-access-t2xsk\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.457566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.457513 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39ab2936-c145-4f75-9549-1e4d47a9d5f6-webhook-cert\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.558750 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.558631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xsk\" (UniqueName: \"kubernetes.io/projected/39ab2936-c145-4f75-9549-1e4d47a9d5f6-kube-api-access-t2xsk\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.559116 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.558772 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39ab2936-c145-4f75-9549-1e4d47a9d5f6-webhook-cert\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.561165 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.561144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39ab2936-c145-4f75-9549-1e4d47a9d5f6-webhook-cert\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.567718 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.567695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xsk\" (UniqueName: \"kubernetes.io/projected/39ab2936-c145-4f75-9549-1e4d47a9d5f6-kube-api-access-t2xsk\") pod \"maas-controller-969c84445-tjqh6\" (UID: \"39ab2936-c145-4f75-9549-1e4d47a9d5f6\") " pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.721544 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.721505 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:57.856541 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:57.856463 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-tjqh6"] Jul 22 01:57:57.859818 ip-10-0-133-122 kubenswrapper[2585]: W0722 01:57:57.859792 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ab2936_c145_4f75_9549_1e4d47a9d5f6.slice/crio-50323c9e5170439f75585949ae9d7f13e5fe1198a249c23bfb04ece68ac47403 WatchSource:0}: Error finding container 50323c9e5170439f75585949ae9d7f13e5fe1198a249c23bfb04ece68ac47403: Status 404 returned error can't find the container with id 50323c9e5170439f75585949ae9d7f13e5fe1198a249c23bfb04ece68ac47403 Jul 22 01:57:58.512342 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:58.512245 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-tjqh6" event={"ID":"39ab2936-c145-4f75-9549-1e4d47a9d5f6","Type":"ContainerStarted","Data":"f3e6a7993830f1b31d9d3c22b915c5c65c7d34465fa5ab67c9fe964c0b7914a6"} Jul 22 01:57:58.512342 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:58.512316 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-tjqh6" event={"ID":"39ab2936-c145-4f75-9549-1e4d47a9d5f6","Type":"ContainerStarted","Data":"50323c9e5170439f75585949ae9d7f13e5fe1198a249c23bfb04ece68ac47403"} Jul 22 01:57:58.512544 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:58.512345 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:57:58.539015 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:57:58.538957 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-969c84445-tjqh6" podStartSLOduration=1.1473088790000001 podStartE2EDuration="1.538938409s" totalImagesPullingTime="391.62953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:58.537116079 +0000 UTC m=+756.437994599" watchObservedRunningTime="2026-07-22 01:57:58.538938409 +0000 UTC m=+756.439816905" Jul 22 01:58:09.519910 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:09.519878 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-969c84445-tjqh6" Jul 22 01:58:31.794030 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:31.793992 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:31.853566 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:31.853528 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:38.215815 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:38.215701 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:38.302748 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:38.302711 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:38.468078 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:38.467973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:44.163575 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:44.163491 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:44.271318 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:44.271284 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:44.365544 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:44.365506 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:44.589100 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:44.589066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:50.186612 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:50.186575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 01:58:50.286180 ip-10-0-133-122 kubenswrapper[2585]: I0722 01:58:50.286144 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:00:28.107157 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:00:28.107118 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:04:22.598438 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:04:22.598402 2585 scope.go:117] "RemoveContainer" containerID="5b0585bc3df11cf90cf4277faba607b9edb4caf9eff3c6f7bf8e45c2fcbb1b31" Jul 22 02:06:25.466457 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:25.466402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:29.318192 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:29.318150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:31.530318 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:31.530272 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:42.100593 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:42.100553 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:42.200474 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:42.200432 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:55.723289 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:55.723250 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:06:57.035077 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:06:57.035031 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:07:44.793134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:07:44.793050 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:09:11.274250 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:09:11.274213 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:03.785852 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:03.785814 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:24.153659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:24.153607 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:27.344974 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:27.344938 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:27.444982 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:27.444949 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:27.872821 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:27.872778 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:33.588512 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:33.588471 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:33.657630 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:33.657592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:33.853819 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:33.853724 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:34.154907 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.154835 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:10:34.158373 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.158356 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.161458 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.161438 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a46a06be-openshift-default-dockercfg-w64tx\"" Jul 22 02:10:34.176123 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.176088 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:34.182702 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.182671 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:10:34.305369 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305319 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8fx\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305556 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.305830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.305765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406629 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406523 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8fx\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406629 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406584 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.406920 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407297 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.406940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407297 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.407069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407297 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.407260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407458 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.407376 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407458 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.407418 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.407862 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.407837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.409654 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.409618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.409749 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.409624 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.416361 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.416331 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.416467 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.416454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8fx\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx\") pod \"e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.469050 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.468997 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:34.585621 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.585585 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:34.621330 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.621304 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:10:34.623746 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:10:34.623716 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e8aa01_a91e_4552_b8bb_924035ccaa65.slice/crio-531b9fddd02b0777efdcd1937013bdef1d7d9418a07169e0ae4b8e4afa853e86 WatchSource:0}: Error finding container 531b9fddd02b0777efdcd1937013bdef1d7d9418a07169e0ae4b8e4afa853e86: Status 404 returned error can't find the container with id 531b9fddd02b0777efdcd1937013bdef1d7d9418a07169e0ae4b8e4afa853e86 Jul 22 02:10:34.625990 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.625960 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:34.626062 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.626024 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:34.626062 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:34.626054 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:35.277159 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:35.277117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerStarted","Data":"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b"} Jul 22 02:10:35.277424 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:35.277178 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerStarted","Data":"531b9fddd02b0777efdcd1937013bdef1d7d9418a07169e0ae4b8e4afa853e86"} Jul 22 02:10:35.298307 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:35.298248 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podStartSLOduration=1.2982296039999999 podStartE2EDuration="1.298229604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:10:35.296747709 +0000 UTC m=+1513.197626231" watchObservedRunningTime="2026-07-22 02:10:35.298229604 +0000 UTC m=+1513.199108101" Jul 22 02:10:35.469514 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:35.469482 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:35.474113 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:35.474083 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:36.281869 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.281840 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:36.282954 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.282935 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:36.660191 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.660098 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:36.760881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.760839 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:36.888143 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.888105 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:10:36.892462 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.892433 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:36.895304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.895280 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a46a06be-serving-cert\"" Jul 22 02:10:36.900191 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.900162 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:10:36.941814 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:36.941782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:37.035474 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.035434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhms\" (UniqueName: \"kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.035674 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.035486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.035674 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.035564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.042329 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.042290 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:37.137101 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.137060 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.137290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.137165 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.137334 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.137294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhms\" (UniqueName: \"kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.137910 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.137880 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.139692 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.139669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.163478 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.163448 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhms\" (UniqueName: \"kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms\") pod \"maas-api-e2e-ait-a46a06be-75f97c79df-z8856\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.206489 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.206398 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:37.374987 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.374961 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:10:37.377363 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:10:37.377337 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c46b26b_76c1_4037_a9c6_e078a299d053.slice/crio-254b2d52b51e80f11eefa84de971d832ce255db3574e11d04cc50f58b99860fc WatchSource:0}: Error finding container 254b2d52b51e80f11eefa84de971d832ce255db3574e11d04cc50f58b99860fc: Status 404 returned error can't find the container with id 254b2d52b51e80f11eefa84de971d832ce255db3574e11d04cc50f58b99860fc Jul 22 02:10:37.378928 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.378907 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:10:37.620446 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.620353 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:37.738728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:37.738689 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:10:38.294188 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:38.294140 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" event={"ID":"0c46b26b-76c1-4037-a9c6-e078a299d053","Type":"ContainerStarted","Data":"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a"} Jul 22 02:10:38.294367 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:38.294331 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:38.294421 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:38.294367 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" event={"ID":"0c46b26b-76c1-4037-a9c6-e078a299d053","Type":"ContainerStarted","Data":"254b2d52b51e80f11eefa84de971d832ce255db3574e11d04cc50f58b99860fc"} Jul 22 02:10:38.313163 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:38.313116 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" podStartSLOduration=1.788790235 podStartE2EDuration="2.31309953s" totalImagesPullingTime="524.309295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:10:38.311598625 +0000 UTC m=+1516.212477123" watchObservedRunningTime="2026-07-22 02:10:38.31309953 +0000 UTC m=+1516.213978026" Jul 22 02:10:44.304940 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:44.304911 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:10:51.774362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:51.773107 2585 patch_prober.go:28] interesting pod/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:37350->10.134.0.42:15021: read: connection reset by peer" start-of-body= Jul 22 02:10:51.774362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:51.773179 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:37350->10.134.0.42:15021: read: connection reset by peer" Jul 22 02:10:52.376657 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.376605 2585 generic.go:356] "Generic (PLEG): container finished" podID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerID="2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b" exitCode=0 Jul 22 02:10:52.376841 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.376676 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerDied","Data":"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b"} Jul 22 02:10:52.377117 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.377099 2585 scope.go:117] "RemoveContainer" containerID="2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b" Jul 22 02:10:52.378131 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.378106 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:52.378194 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.378171 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:52.378231 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:52.378220 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:10:53.384150 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:53.384113 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerStarted","Data":"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a"} Jul 22 02:10:53.469604 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:53.469566 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:53.469947 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:53.469926 2585 patch_prober.go:28] interesting pod/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 02:10:53.470000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:53.469972 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 02:10:54.469907 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:54.469869 2585 patch_prober.go:28] interesting pod/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 02:10:54.470326 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:54.469923 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 02:10:55.470413 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:55.470370 2585 patch_prober.go:28] interesting pod/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 02:10:55.470940 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:55.470436 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 02:10:56.470305 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:56.470261 2585 patch_prober.go:28] interesting pod/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 02:10:56.470501 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:56.470326 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 02:10:57.472991 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:57.472956 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:57.473392 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:57.473216 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:10:57.474027 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:10:57.474007 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:11:22.800306 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:22.800261 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:11:22.800989 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:22.800484 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" podUID="0c46b26b-76c1-4037-a9c6-e078a299d053" containerName="maas-api" containerID="cri-o://5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a" gracePeriod=30 Jul 22 02:11:22.830062 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:22.830023 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:23.065165 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.065099 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:11:23.124690 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.124617 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls\") pod \"0c46b26b-76c1-4037-a9c6-e078a299d053\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " Jul 22 02:11:23.124891 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.124732 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjhms\" (UniqueName: \"kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms\") pod \"0c46b26b-76c1-4037-a9c6-e078a299d053\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " Jul 22 02:11:23.124891 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.124797 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca\") pod \"0c46b26b-76c1-4037-a9c6-e078a299d053\" (UID: \"0c46b26b-76c1-4037-a9c6-e078a299d053\") " Jul 22 02:11:23.125161 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.125132 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c46b26b-76c1-4037-a9c6-e078a299d053" (UID: "0c46b26b-76c1-4037-a9c6-e078a299d053"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:11:23.126965 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.126944 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c46b26b-76c1-4037-a9c6-e078a299d053" (UID: "0c46b26b-76c1-4037-a9c6-e078a299d053"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:11:23.127031 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.126984 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms" (OuterVolumeSpecName: "kube-api-access-kjhms") pod "0c46b26b-76c1-4037-a9c6-e078a299d053" (UID: "0c46b26b-76c1-4037-a9c6-e078a299d053"). InnerVolumeSpecName "kube-api-access-kjhms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:11:23.226004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.225955 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c46b26b-76c1-4037-a9c6-e078a299d053-openshift-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:23.226004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.225993 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c46b26b-76c1-4037-a9c6-e078a299d053-maas-api-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:23.226004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.226006 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjhms\" (UniqueName: \"kubernetes.io/projected/0c46b26b-76c1-4037-a9c6-e078a299d053-kube-api-access-kjhms\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:23.539925 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.539888 2585 generic.go:356] "Generic (PLEG): container finished" podID="0c46b26b-76c1-4037-a9c6-e078a299d053" containerID="5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a" exitCode=0 Jul 22 02:11:23.540107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.539936 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" Jul 22 02:11:23.540107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.539976 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" event={"ID":"0c46b26b-76c1-4037-a9c6-e078a299d053","Type":"ContainerDied","Data":"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a"} Jul 22 02:11:23.540107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.540033 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856" event={"ID":"0c46b26b-76c1-4037-a9c6-e078a299d053","Type":"ContainerDied","Data":"254b2d52b51e80f11eefa84de971d832ce255db3574e11d04cc50f58b99860fc"} Jul 22 02:11:23.540107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.540063 2585 scope.go:117] "RemoveContainer" containerID="5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a" Jul 22 02:11:23.549390 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.549377 2585 scope.go:117] "RemoveContainer" containerID="5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a" Jul 22 02:11:23.549696 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:23.549675 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a\": container with ID starting with 5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a not found: ID does not exist" containerID="5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a" Jul 22 02:11:23.549802 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.549706 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a"} err="failed to get container status \"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a\": rpc error: code = NotFound desc = could not find container \"5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a\": container with ID starting with 5af4583138809f5c0589e4830af39452adda694ff4bf3ab0391b905f7a26f03a not found: ID does not exist" Jul 22 02:11:23.561845 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.561818 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:11:23.567625 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:23.567599 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a46a06be-75f97c79df-z8856"] Jul 22 02:11:24.677983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:24.677951 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c46b26b-76c1-4037-a9c6-e078a299d053" path="/var/lib/kubelet/pods/0c46b26b-76c1-4037-a9c6-e078a299d053/volumes" Jul 22 02:11:27.416385 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:27.416348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:27.562737 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:27.562704 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:11:27.563222 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:27.563186 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" containerID="cri-o://0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a" gracePeriod=30 Jul 22 02:11:28.661941 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.661904 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:11:28.662428 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.662387 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c46b26b-76c1-4037-a9c6-e078a299d053" containerName="maas-api" Jul 22 02:11:28.662428 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.662411 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c46b26b-76c1-4037-a9c6-e078a299d053" containerName="maas-api" Jul 22 02:11:28.662677 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.662507 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c46b26b-76c1-4037-a9c6-e078a299d053" containerName="maas-api" Jul 22 02:11:28.667302 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.667282 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.670461 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.670443 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1e59c61a-gw-openshift-default-dockercfg-2hjfq\"" Jul 22 02:11:28.680341 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.680315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:11:28.735804 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.735768 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:28.779337 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779596 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnf2\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779755 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779823 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779823 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779934 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.779934 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779912 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.780028 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.779969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.780154 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.780082 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.880916 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.880865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.880960 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.880982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnf2\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.880998 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881454 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881454 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881454 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881432 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881616 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881514 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881616 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881736 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.881943 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.881924 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.883438 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.883416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.883650 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.883616 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.889766 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.889740 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.889925 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.889909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnf2\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2\") pod \"e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:28.979124 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:28.979083 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:29.121270 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.121244 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:11:29.123497 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:11:29.123466 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29653566_3857_45f1_bf21_5b87d357150f.slice/crio-66abc8b85e7e8e6d4bc8568c02e475a3b552d761bee9d56b68cabf78d3e75f29 WatchSource:0}: Error finding container 66abc8b85e7e8e6d4bc8568c02e475a3b552d761bee9d56b68cabf78d3e75f29: Status 404 returned error can't find the container with id 66abc8b85e7e8e6d4bc8568c02e475a3b552d761bee9d56b68cabf78d3e75f29 Jul 22 02:11:29.125618 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.125585 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:29.125730 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.125675 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:29.125730 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.125711 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:29.315906 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.315797 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:29.574326 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.574222 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerStarted","Data":"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf"} Jul 22 02:11:29.574326 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.574282 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerStarted","Data":"66abc8b85e7e8e6d4bc8568c02e475a3b552d761bee9d56b68cabf78d3e75f29"} Jul 22 02:11:29.612434 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.612373 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podStartSLOduration=1.6123546869999998 podStartE2EDuration="1.612354687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:11:29.609841794 +0000 UTC m=+1567.510720312" watchObservedRunningTime="2026-07-22 02:11:29.612354687 +0000 UTC m=+1567.513233183" Jul 22 02:11:29.979290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:29.979250 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:30.984896 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:30.984858 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:31.104188 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.104148 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:31.197277 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.197243 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:11:31.207014 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.206989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.210953 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.210927 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1e59c61a-serving-cert\"" Jul 22 02:11:31.212782 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.212756 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:11:31.235353 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.235251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:31.306482 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.306443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.306694 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.306489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8hc\" (UniqueName: \"kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.306694 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.306532 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.407121 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.407077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.407327 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.407203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.407327 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:31.407228 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-serving-cert: secret "maas-api-e2e-migrate-1e59c61a-serving-cert" not found Jul 22 02:11:31.407327 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:31.407292 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls podName:20f1623a-4130-4822-82ca-589547777ab5 nodeName:}" failed. No retries permitted until 2026-07-22 02:11:31.907275174 +0000 UTC m=+1569.808153648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls") pod "maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" (UID: "20f1623a-4130-4822-82ca-589547777ab5") : secret "maas-api-e2e-migrate-1e59c61a-serving-cert" not found Jul 22 02:11:31.407327 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.407229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8hc\" (UniqueName: \"kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.407967 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.407944 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.417280 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.417253 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8hc\" (UniqueName: \"kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.584540 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.584455 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:31.585428 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.585409 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:31.823362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.823327 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:31.912745 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.912616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:31.912921 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:31.912777 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-serving-cert: secret "maas-api-e2e-migrate-1e59c61a-serving-cert" not found Jul 22 02:11:31.912921 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:31.912840 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls podName:20f1623a-4130-4822-82ca-589547777ab5 nodeName:}" failed. No retries permitted until 2026-07-22 02:11:32.912822064 +0000 UTC m=+1570.813700556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls") pod "maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" (UID: "20f1623a-4130-4822-82ca-589547777ab5") : secret "maas-api-e2e-migrate-1e59c61a-serving-cert" not found Jul 22 02:11:31.924530 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:31.924492 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:32.025362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.025316 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:11:32.829963 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.829898 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:11:32.923625 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923588 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.923625 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923624 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.923909 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923673 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.923909 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923691 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp8fx\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.923909 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923718 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.923909 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923992 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.923882 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924009 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924050 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924090 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data\") pod \"97e8aa01-a91e-4552-b8bb-924035ccaa65\" (UID: \"97e8aa01-a91e-4552-b8bb-924035ccaa65\") " Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924099 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:32.924116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924107 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:11:32.924468 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:32.924616 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924597 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:32.924719 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924622 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:32.924719 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924653 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e8aa01-a91e-4552-b8bb-924035ccaa65-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:32.924719 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924667 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:32.924986 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.924955 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data" (OuterVolumeSpecName: "istio-data") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:32.926309 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.926286 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:11:32.926485 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.926461 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:11:32.926547 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.926471 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx" (OuterVolumeSpecName: "kube-api-access-sp8fx") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "kube-api-access-sp8fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:11:32.926599 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.926569 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token" (OuterVolumeSpecName: "istio-token") pod "97e8aa01-a91e-4552-b8bb-924035ccaa65" (UID: "97e8aa01-a91e-4552-b8bb-924035ccaa65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:11:32.927140 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:32.927121 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:33.021545 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.021508 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:33.026242 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.026214 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:33.026544 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.026248 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:33.026544 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.026276 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8fx\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-kube-api-access-sp8fx\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:33.026544 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.026291 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:33.026544 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.026303 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e8aa01-a91e-4552-b8bb-924035ccaa65-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:11:33.163814 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.163786 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:11:33.165231 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:11:33.165207 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f1623a_4130_4822_82ca_589547777ab5.slice/crio-e4d8d5e3cf7192e77e37892d144e37107d3bacc97487eace681c34de65866ef7 WatchSource:0}: Error finding container e4d8d5e3cf7192e77e37892d144e37107d3bacc97487eace681c34de65866ef7: Status 404 returned error can't find the container with id e4d8d5e3cf7192e77e37892d144e37107d3bacc97487eace681c34de65866ef7 Jul 22 02:11:33.597198 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.597167 2585 generic.go:356] "Generic (PLEG): container finished" podID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerID="0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a" exitCode=0 Jul 22 02:11:33.597388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.597235 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" Jul 22 02:11:33.597388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.597255 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerDied","Data":"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a"} Jul 22 02:11:33.597388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.597341 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh" event={"ID":"97e8aa01-a91e-4552-b8bb-924035ccaa65","Type":"ContainerDied","Data":"531b9fddd02b0777efdcd1937013bdef1d7d9418a07169e0ae4b8e4afa853e86"} Jul 22 02:11:33.597388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.597371 2585 scope.go:117] "RemoveContainer" containerID="0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a" Jul 22 02:11:33.599650 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.599610 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" event={"ID":"20f1623a-4130-4822-82ca-589547777ab5","Type":"ContainerStarted","Data":"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1"} Jul 22 02:11:33.599768 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.599691 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" event={"ID":"20f1623a-4130-4822-82ca-589547777ab5","Type":"ContainerStarted","Data":"e4d8d5e3cf7192e77e37892d144e37107d3bacc97487eace681c34de65866ef7"} Jul 22 02:11:33.599768 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.599741 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:33.608703 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.608684 2585 scope.go:117] "RemoveContainer" containerID="2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b" Jul 22 02:11:33.617538 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.617516 2585 scope.go:117] "RemoveContainer" containerID="0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a" Jul 22 02:11:33.617856 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:33.617830 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a\": container with ID starting with 0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a not found: ID does not exist" containerID="0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a" Jul 22 02:11:33.617926 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.617872 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a"} err="failed to get container status \"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a\": rpc error: code = NotFound desc = could not find container \"0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a\": container with ID starting with 0667c8d3afa80b134d6b7f176214adf2612eff4d51a2ede51dc077105fb5dc6a not found: ID does not exist" Jul 22 02:11:33.617926 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.617898 2585 scope.go:117] "RemoveContainer" containerID="2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b" Jul 22 02:11:33.618144 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:11:33.618123 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b\": container with ID starting with 2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b not found: ID does not exist" containerID="2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b" Jul 22 02:11:33.618237 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.618149 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b"} err="failed to get container status \"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b\": rpc error: code = NotFound desc = could not find container \"2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b\": container with ID starting with 2a6dc09d0de608430791c2a1fcc52805fc7b650e1e0ce00d8804862f1e78af5b not found: ID does not exist" Jul 22 02:11:33.624193 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.624139 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" podStartSLOduration=2.326320318 podStartE2EDuration="2.6241207s" totalImagesPullingTime="297.800382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:11:33.621454805 +0000 UTC m=+1571.522333303" watchObservedRunningTime="2026-07-22 02:11:33.6241207 +0000 UTC m=+1571.524999198" Jul 22 02:11:33.636601 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.636568 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:11:33.642245 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:33.642217 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a46a06be-openshift-default-6b4df846fc-r46bh"] Jul 22 02:11:34.678057 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:34.678024 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" path="/var/lib/kubelet/pods/97e8aa01-a91e-4552-b8bb-924035ccaa65/volumes" Jul 22 02:11:39.611140 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:39.611113 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:11:43.673966 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.673932 2585 generic.go:356] "Generic (PLEG): container finished" podID="29653566-3857-45f1-bf21-5b87d357150f" containerID="e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf" exitCode=0 Jul 22 02:11:43.674509 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.674013 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerDied","Data":"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf"} Jul 22 02:11:43.674579 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.674535 2585 scope.go:117] "RemoveContainer" containerID="e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf" Jul 22 02:11:43.675681 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.675629 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:43.675766 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.675727 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:43.675807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:43.675763 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:11:44.681141 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:44.681100 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerStarted","Data":"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de"} Jul 22 02:11:44.980019 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:44.979920 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:44.981200 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:44.981171 2585 patch_prober.go:28] interesting pod/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 22 02:11:44.981348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:44.981254 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 22 02:11:45.980251 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:45.980212 2585 patch_prober.go:28] interesting pod/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 22 02:11:45.980687 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:45.980268 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 22 02:11:46.585008 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:46.584972 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:46.980423 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:46.980381 2585 patch_prober.go:28] interesting pod/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 22 02:11:46.980847 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:46.980446 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 22 02:11:47.980049 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:47.980003 2585 patch_prober.go:28] interesting pod/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 22 02:11:47.980241 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:47.980068 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 22 02:11:48.983625 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:48.983590 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:11:48.984731 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:11:48.984713 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:12:06.739968 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:06.739932 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:12:06.740536 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:06.740133 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" podUID="20f1623a-4130-4822-82ca-589547777ab5" containerName="maas-api" containerID="cri-o://5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1" gracePeriod=30 Jul 22 02:12:06.769134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:06.769077 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:07.001604 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.001534 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:12:07.070136 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.070100 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca\") pod \"20f1623a-4130-4822-82ca-589547777ab5\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " Jul 22 02:12:07.070300 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.070227 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") pod \"20f1623a-4130-4822-82ca-589547777ab5\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " Jul 22 02:12:07.070300 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.070265 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm8hc\" (UniqueName: \"kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc\") pod \"20f1623a-4130-4822-82ca-589547777ab5\" (UID: \"20f1623a-4130-4822-82ca-589547777ab5\") " Jul 22 02:12:07.070498 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.070471 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20f1623a-4130-4822-82ca-589547777ab5" (UID: "20f1623a-4130-4822-82ca-589547777ab5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:12:07.070567 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.070553 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f1623a-4130-4822-82ca-589547777ab5-openshift-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:07.072451 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.072423 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20f1623a-4130-4822-82ca-589547777ab5" (UID: "20f1623a-4130-4822-82ca-589547777ab5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:12:07.072596 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.072467 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc" (OuterVolumeSpecName: "kube-api-access-cm8hc") pod "20f1623a-4130-4822-82ca-589547777ab5" (UID: "20f1623a-4130-4822-82ca-589547777ab5"). InnerVolumeSpecName "kube-api-access-cm8hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:12:07.172000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.171964 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f1623a-4130-4822-82ca-589547777ab5-maas-api-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:07.172000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.171993 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8hc\" (UniqueName: \"kubernetes.io/projected/20f1623a-4130-4822-82ca-589547777ab5-kube-api-access-cm8hc\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:07.805159 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.805121 2585 generic.go:356] "Generic (PLEG): container finished" podID="20f1623a-4130-4822-82ca-589547777ab5" containerID="5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1" exitCode=0 Jul 22 02:12:07.805728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.805187 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" Jul 22 02:12:07.805728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.805209 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" event={"ID":"20f1623a-4130-4822-82ca-589547777ab5","Type":"ContainerDied","Data":"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1"} Jul 22 02:12:07.805728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.805266 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm" event={"ID":"20f1623a-4130-4822-82ca-589547777ab5","Type":"ContainerDied","Data":"e4d8d5e3cf7192e77e37892d144e37107d3bacc97487eace681c34de65866ef7"} Jul 22 02:12:07.805728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.805288 2585 scope.go:117] "RemoveContainer" containerID="5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1" Jul 22 02:12:07.814368 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.814351 2585 scope.go:117] "RemoveContainer" containerID="5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1" Jul 22 02:12:07.814626 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:12:07.814607 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1\": container with ID starting with 5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1 not found: ID does not exist" containerID="5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1" Jul 22 02:12:07.814721 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.814662 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1"} err="failed to get container status \"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1\": rpc error: code = NotFound desc = could not find container \"5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1\": container with ID starting with 5fc28a6a06fe0f823c6424676868edb8c9fe8a1ee6b65c1cf2e5c90e747854c1 not found: ID does not exist" Jul 22 02:12:07.829089 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.829056 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:12:07.833696 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:07.833671 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1e59c61a-f557574cd-v86dm"] Jul 22 02:12:08.677027 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:08.676996 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f1623a-4130-4822-82ca-589547777ab5" path="/var/lib/kubelet/pods/20f1623a-4130-4822-82ca-589547777ab5/volumes" Jul 22 02:12:11.271831 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.271791 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:11.419039 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.419003 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:12:11.419305 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.419255 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" containerID="cri-o://47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de" gracePeriod=30 Jul 22 02:12:11.436249 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.436217 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:11.735499 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.735466 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:11.974430 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:11.974396 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:12.300658 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.300614 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:12:12.301120 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301104 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301120 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301121 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301196 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301135 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301196 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301140 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301196 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301147 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f1623a-4130-4822-82ca-589547777ab5" containerName="maas-api" Jul 22 02:12:12.301196 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301153 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f1623a-4130-4822-82ca-589547777ab5" containerName="maas-api" Jul 22 02:12:12.301322 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301227 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301322 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301235 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e8aa01-a91e-4552-b8bb-924035ccaa65" containerName="istio-proxy" Jul 22 02:12:12.301322 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.301241 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f1623a-4130-4822-82ca-589547777ab5" containerName="maas-api" Jul 22 02:12:12.305659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.305627 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.309289 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.309267 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-68767aef-openshift-default-dockercfg-2rrdr\"" Jul 22 02:12:12.318215 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.318189 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:12:12.337300 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.337264 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:12.426486 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426450 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426679 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426501 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqps\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426679 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426679 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426797 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426797 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426749 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426864 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426864 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.426926 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.426873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527486 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527449 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527486 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527497 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527762 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527520 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527762 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527762 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527762 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.527762 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527677 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528016 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528016 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqps\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528016 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527942 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528016 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.527989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528199 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.528052 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528296 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.528276 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.528532 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.528516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.530018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.529996 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.530185 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.530170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.536815 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.536789 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.536935 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.536888 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqps\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps\") pod \"e2e-ait-68767aef-openshift-default-6486686b47-v2g9c\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.616469 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.616382 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:12.718013 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.717922 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:12.770988 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.770961 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:12:12.773302 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:12:12.773275 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecfa400_742a_4927_b0ce_3042bd68c01d.slice/crio-dabca489f96dbae76ae16c0971719baf4d2358a8f86d4a1616d3793369fcefef WatchSource:0}: Error finding container dabca489f96dbae76ae16c0971719baf4d2358a8f86d4a1616d3793369fcefef: Status 404 returned error can't find the container with id dabca489f96dbae76ae16c0971719baf4d2358a8f86d4a1616d3793369fcefef Jul 22 02:12:12.775493 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.775459 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:12.775574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.775523 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:12.775574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.775550 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:12.831620 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:12.831585 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerStarted","Data":"dabca489f96dbae76ae16c0971719baf4d2358a8f86d4a1616d3793369fcefef"} Jul 22 02:12:13.838197 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:13.838159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerStarted","Data":"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca"} Jul 22 02:12:13.863579 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:13.863483 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podStartSLOduration=1.863465643 podStartE2EDuration="1.863465643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:12:13.861013944 +0000 UTC m=+1611.761892482" watchObservedRunningTime="2026-07-22 02:12:13.863465643 +0000 UTC m=+1611.764344140" Jul 22 02:12:14.616924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:14.616884 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:14.622485 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:14.622456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:14.843589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:14.843553 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:14.844566 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:14.844549 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:15.215933 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:15.215894 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:15.312088 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:15.312040 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:15.379154 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:15.379108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:15.480763 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:15.480661 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:15.899840 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:15.899738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:16.003094 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.003037 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:16.693835 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.693768 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:12:16.772874 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.772843 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcnf2\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773088 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.772909 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773088 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773026 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773088 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773072 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773265 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773103 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773265 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773265 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773235 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773280 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773307 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy\") pod \"29653566-3857-45f1-bf21-5b87d357150f\" (UID: \"29653566-3857-45f1-bf21-5b87d357150f\") " Jul 22 02:12:16.773418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773380 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:12:16.773418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773390 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:12:16.773418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773384 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:12:16.773695 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.773578 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:12:16.774178 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.774150 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.774342 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.774182 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29653566-3857-45f1-bf21-5b87d357150f-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.774342 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.774199 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.774342 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.774214 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.774526 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.774352 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data" (OuterVolumeSpecName: "istio-data") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:12:16.775399 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.775373 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:12:16.775485 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.775400 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2" (OuterVolumeSpecName: "kube-api-access-vcnf2") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "kube-api-access-vcnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:12:16.775574 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.775558 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:12:16.775624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.775601 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token" (OuterVolumeSpecName: "istio-token") pod "29653566-3857-45f1-bf21-5b87d357150f" (UID: "29653566-3857-45f1-bf21-5b87d357150f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:12:16.856348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.856314 2585 generic.go:356] "Generic (PLEG): container finished" podID="29653566-3857-45f1-bf21-5b87d357150f" containerID="47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de" exitCode=0 Jul 22 02:12:16.856520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.856365 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" Jul 22 02:12:16.856520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.856403 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerDied","Data":"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de"} Jul 22 02:12:16.856520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.856459 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z" event={"ID":"29653566-3857-45f1-bf21-5b87d357150f","Type":"ContainerDied","Data":"66abc8b85e7e8e6d4bc8568c02e475a3b552d761bee9d56b68cabf78d3e75f29"} Jul 22 02:12:16.856520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.856483 2585 scope.go:117] "RemoveContainer" containerID="47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de" Jul 22 02:12:16.866625 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.866607 2585 scope.go:117] "RemoveContainer" containerID="e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf" Jul 22 02:12:16.875071 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875042 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnf2\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-kube-api-access-vcnf2\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.875071 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875074 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29653566-3857-45f1-bf21-5b87d357150f-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.875288 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875088 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29653566-3857-45f1-bf21-5b87d357150f-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.875288 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875102 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.875288 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875114 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29653566-3857-45f1-bf21-5b87d357150f-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:12:16.875850 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.875837 2585 scope.go:117] "RemoveContainer" containerID="47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de" Jul 22 02:12:16.876105 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:12:16.876087 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de\": container with ID starting with 47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de not found: ID does not exist" containerID="47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de" Jul 22 02:12:16.876161 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.876122 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de"} err="failed to get container status \"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de\": rpc error: code = NotFound desc = could not find container \"47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de\": container with ID starting with 47c0afd1881e1fd7f6a4a985f5548de9b9ab09da2c9986b40f7a0b7708f4a9de not found: ID does not exist" Jul 22 02:12:16.876161 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.876140 2585 scope.go:117] "RemoveContainer" containerID="e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf" Jul 22 02:12:16.876381 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:12:16.876367 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf\": container with ID starting with e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf not found: ID does not exist" containerID="e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf" Jul 22 02:12:16.876424 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.876385 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf"} err="failed to get container status \"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf\": rpc error: code = NotFound desc = could not find container \"e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf\": container with ID starting with e2fcc0e081c26ca2e67fed21fe9b239252db28042a1c37cfadbdf65799adfabf not found: ID does not exist" Jul 22 02:12:16.880225 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.880202 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:12:16.885134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:16.885113 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1e59c61a-gw-openshift-default-6975d85b4c-lgt9z"] Jul 22 02:12:18.678083 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:18.678046 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29653566-3857-45f1-bf21-5b87d357150f" path="/var/lib/kubelet/pods/29653566-3857-45f1-bf21-5b87d357150f/volumes" Jul 22 02:12:28.922892 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.922857 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerID="1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca" exitCode=0 Jul 22 02:12:28.923364 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.922906 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerDied","Data":"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca"} Jul 22 02:12:28.923364 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.923295 2585 scope.go:117] "RemoveContainer" containerID="1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca" Jul 22 02:12:28.924362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.924335 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:28.924423 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.924404 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:28.924463 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:28.924435 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:29.844661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:29.843762 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:29.933783 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:29.933740 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerStarted","Data":"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3"} Jul 22 02:12:30.617583 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:30.617541 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:30.618815 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:30.618787 2585 patch_prober.go:28] interesting pod/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 02:12:30.618979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:30.618837 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 02:12:31.617733 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:31.617696 2585 patch_prober.go:28] interesting pod/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 02:12:31.618118 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:31.617750 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 02:12:32.616956 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:32.616913 2585 patch_prober.go:28] interesting pod/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 02:12:32.617134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:32.616973 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 02:12:33.617082 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:33.617040 2585 patch_prober.go:28] interesting pod/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 02:12:33.617487 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:33.617100 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 02:12:34.620950 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:34.620916 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:34.621346 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:34.621209 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:34.621993 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:34.621970 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:12:51.980121 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:51.980086 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:56.426830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:56.426792 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:57.812837 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:57.812795 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:12:57.813423 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:57.813083 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" containerID="cri-o://fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3" gracePeriod=30 Jul 22 02:12:57.827006 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:57.826970 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:58.067707 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.067593 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:58.321101 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.321010 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:58.655707 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.655611 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:12:58.656279 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656262 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.656376 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656284 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.656376 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656302 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.656376 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656311 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.656551 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656413 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.656551 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.656425 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="29653566-3857-45f1-bf21-5b87d357150f" containerName="istio-proxy" Jul 22 02:12:58.659758 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.659739 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.663189 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.663167 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4735213f-openshift-default-dockercfg-77lxc\"" Jul 22 02:12:58.686468 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.686428 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:12:58.725613 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.725559 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:58.814992 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.814956 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.814997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdkl\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815103 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815240 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.815626 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.815433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.916609 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.916781 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.916781 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.916781 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdkl\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.916781 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917045 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917045 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917045 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917045 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.916916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.917037 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.917063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.917108 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.917165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.917672 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.917626 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.919005 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.918975 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.919247 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.919231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.930859 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.930836 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.932477 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.932460 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdkl\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl\") pod \"e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:58.970134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:58.970099 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:12:59.062447 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:59.061916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:12:59.120875 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:59.120849 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:12:59.123335 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:12:59.123308 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d53179_bad2_4258_8b86_52152eeca379.slice/crio-03dc78c666a4dcdb24cd9f3def79d24ee12e26649b666986c329d2f44c68ec5f WatchSource:0}: Error finding container 03dc78c666a4dcdb24cd9f3def79d24ee12e26649b666986c329d2f44c68ec5f: Status 404 returned error can't find the container with id 03dc78c666a4dcdb24cd9f3def79d24ee12e26649b666986c329d2f44c68ec5f Jul 22 02:12:59.125503 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:59.125471 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:59.125589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:59.125535 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:12:59.125589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:12:59.125563 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:13:00.091612 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:00.091575 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerStarted","Data":"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16"} Jul 22 02:13:00.092059 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:00.091685 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerStarted","Data":"03dc78c666a4dcdb24cd9f3def79d24ee12e26649b666986c329d2f44c68ec5f"} Jul 22 02:13:00.115676 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:00.115597 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podStartSLOduration=2.11557666 podStartE2EDuration="2.11557666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:13:00.113969285 +0000 UTC m=+1658.014847782" watchObservedRunningTime="2026-07-22 02:13:00.11557666 +0000 UTC m=+1658.016455156" Jul 22 02:13:00.970472 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:00.970431 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:00.975478 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:00.975451 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:01.096389 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.096349 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:01.097218 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.097200 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:01.504676 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.504600 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:01.605392 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.605357 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:01.702388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.702346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:01.982891 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:01.982850 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:02.186156 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:02.186127 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:02.286683 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:02.286578 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:03.076987 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.076927 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:13:03.109187 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.108698 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerID="fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3" exitCode=0 Jul 22 02:13:03.109187 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.108762 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" Jul 22 02:13:03.109187 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.108798 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerDied","Data":"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3"} Jul 22 02:13:03.109187 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.108844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c" event={"ID":"1ecfa400-742a-4927-b0ce-3042bd68c01d","Type":"ContainerDied","Data":"dabca489f96dbae76ae16c0971719baf4d2358a8f86d4a1616d3793369fcefef"} Jul 22 02:13:03.109187 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.108877 2585 scope.go:117] "RemoveContainer" containerID="fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3" Jul 22 02:13:03.118642 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.118615 2585 scope.go:117] "RemoveContainer" containerID="1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca" Jul 22 02:13:03.129265 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.129238 2585 scope.go:117] "RemoveContainer" containerID="fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3" Jul 22 02:13:03.129610 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:13:03.129588 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3\": container with ID starting with fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3 not found: ID does not exist" containerID="fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3" Jul 22 02:13:03.129706 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.129621 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3"} err="failed to get container status \"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3\": rpc error: code = NotFound desc = could not find container \"fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3\": container with ID starting with fb9191b7f1273a5bca0c0b5ecc8dd1c097983fbd996bb996fe39b0582fe4a7e3 not found: ID does not exist" Jul 22 02:13:03.129706 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.129657 2585 scope.go:117] "RemoveContainer" containerID="1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca" Jul 22 02:13:03.129970 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:13:03.129952 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca\": container with ID starting with 1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca not found: ID does not exist" containerID="1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca" Jul 22 02:13:03.130030 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.129980 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca"} err="failed to get container status \"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca\": rpc error: code = NotFound desc = could not find container \"1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca\": container with ID starting with 1301cd971c7a4c1c8813db23a7ebf9e0e18f47d310f5f293c18b659aec41f2ca not found: ID does not exist" Jul 22 02:13:03.262783 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262740 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262820 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262862 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262908 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wqps\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262937 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262966 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.262997 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.263035 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.263099 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data\") pod \"1ecfa400-742a-4927-b0ce-3042bd68c01d\" (UID: \"1ecfa400-742a-4927-b0ce-3042bd68c01d\") " Jul 22 02:13:03.263380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.263241 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:03.265018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.264002 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:03.265018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.264058 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.265018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.264104 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:03.265018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.264108 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:13:03.265018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.264448 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:03.266048 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.266019 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:13:03.266048 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.266033 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:03.266216 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.266067 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:13:03.266216 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.266073 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps" (OuterVolumeSpecName: "kube-api-access-4wqps") pod "1ecfa400-742a-4927-b0ce-3042bd68c01d" (UID: "1ecfa400-742a-4927-b0ce-3042bd68c01d"). InnerVolumeSpecName "kube-api-access-4wqps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365517 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365563 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365573 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqps\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-kube-api-access-4wqps\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365582 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365590 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365598 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365605 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfa400-742a-4927-b0ce-3042bd68c01d-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.365624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.365613 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfa400-742a-4927-b0ce-3042bd68c01d-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:03.434109 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.434070 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:13:03.437679 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:03.437650 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-68767aef-openshift-default-6486686b47-v2g9c"] Jul 22 02:13:04.677692 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:04.677657 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" path="/var/lib/kubelet/pods/1ecfa400-742a-4927-b0ce-3042bd68c01d/volumes" Jul 22 02:13:15.184091 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.184055 2585 generic.go:356] "Generic (PLEG): container finished" podID="26d53179-bad2-4258-8b86-52152eeca379" containerID="45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16" exitCode=0 Jul 22 02:13:15.184551 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.184131 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerDied","Data":"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16"} Jul 22 02:13:15.184608 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.184581 2585 scope.go:117] "RemoveContainer" containerID="45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16" Jul 22 02:13:15.189292 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.189259 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:13:15.189423 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.189333 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:13:15.189423 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:15.189371 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:13:16.097046 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:16.096990 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:16.191097 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:16.191059 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerStarted","Data":"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702"} Jul 22 02:13:16.971047 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:16.971011 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:16.972248 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:16.972222 2585 patch_prober.go:28] interesting pod/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 02:13:16.972375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:16.972273 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 02:13:17.971444 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:17.971403 2585 patch_prober.go:28] interesting pod/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 02:13:17.971919 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:17.971474 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 02:13:18.971228 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:18.971189 2585 patch_prober.go:28] interesting pod/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 02:13:18.971417 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:18.971261 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 02:13:19.970719 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:19.970679 2585 patch_prober.go:28] interesting pod/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 22 02:13:19.971137 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:19.970741 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 22 02:13:20.975585 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:20.975552 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:20.976038 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:20.975851 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:20.976819 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:20.976802 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:20.982519 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:20.982484 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:21.084410 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:21.084370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:23.009117 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:23.009020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:23.109594 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:23.109554 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:37.070980 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:37.070938 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:41.590739 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:41.590698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:41.759975 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:41.759934 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:13:41.760234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:41.760204 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" containerID="cri-o://c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702" gracePeriod=30 Jul 22 02:13:41.786758 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:41.786696 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:42.049896 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:42.049847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:42.544018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:42.543978 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:42.647710 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:42.647670 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:42.751944 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:42.751907 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:44.263315 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:44.263267 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:47.024995 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.024930 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:47.127721 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127687 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127721 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127730 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127754 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127785 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127811 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127837 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127918 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.127970 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127947 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgdkl\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.128290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.127983 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token\") pod \"26d53179-bad2-4258-8b86-52152eeca379\" (UID: \"26d53179-bad2-4258-8b86-52152eeca379\") " Jul 22 02:13:47.128290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128067 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:47.128290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128114 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:47.128290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:47.128494 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128401 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.128494 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128424 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.128494 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128437 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.128685 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128587 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:13:47.128803 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.128779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data" (OuterVolumeSpecName: "istio-data") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:47.130173 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.130150 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl" (OuterVolumeSpecName: "kube-api-access-wgdkl") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "kube-api-access-wgdkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:13:47.130274 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.130188 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:13:47.130274 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.130188 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:13:47.130274 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.130263 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token" (OuterVolumeSpecName: "istio-token") pod "26d53179-bad2-4258-8b86-52152eeca379" (UID: "26d53179-bad2-4258-8b86-52152eeca379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:13:47.229866 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229827 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.229866 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229859 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d53179-bad2-4258-8b86-52152eeca379-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.229866 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229868 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d53179-bad2-4258-8b86-52152eeca379-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.229866 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229877 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d53179-bad2-4258-8b86-52152eeca379-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.230148 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229887 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdkl\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-kube-api-access-wgdkl\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.230148 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.229895 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d53179-bad2-4258-8b86-52152eeca379-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:13:47.356432 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.356346 2585 generic.go:356] "Generic (PLEG): container finished" podID="26d53179-bad2-4258-8b86-52152eeca379" containerID="c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702" exitCode=0 Jul 22 02:13:47.356432 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.356392 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" Jul 22 02:13:47.356619 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.356427 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerDied","Data":"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702"} Jul 22 02:13:47.356619 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.356493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx" event={"ID":"26d53179-bad2-4258-8b86-52152eeca379","Type":"ContainerDied","Data":"03dc78c666a4dcdb24cd9f3def79d24ee12e26649b666986c329d2f44c68ec5f"} Jul 22 02:13:47.356619 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.356517 2585 scope.go:117] "RemoveContainer" containerID="c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702" Jul 22 02:13:47.366867 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.366844 2585 scope.go:117] "RemoveContainer" containerID="45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16" Jul 22 02:13:47.375489 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.375466 2585 scope.go:117] "RemoveContainer" containerID="c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702" Jul 22 02:13:47.375910 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:13:47.375883 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702\": container with ID starting with c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702 not found: ID does not exist" containerID="c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702" Jul 22 02:13:47.376047 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.375916 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702"} err="failed to get container status \"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702\": rpc error: code = NotFound desc = could not find container \"c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702\": container with ID starting with c6894130690e9d261de131325cf07d6fad2129725614d28e3792edc879c2e702 not found: ID does not exist" Jul 22 02:13:47.376047 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.375935 2585 scope.go:117] "RemoveContainer" containerID="45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16" Jul 22 02:13:47.376267 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:13:47.376248 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16\": container with ID starting with 45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16 not found: ID does not exist" containerID="45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16" Jul 22 02:13:47.376336 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.376276 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16"} err="failed to get container status \"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16\": rpc error: code = NotFound desc = could not find container \"45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16\": container with ID starting with 45b47b200ab0376db9ffba97e0f2220d875e10686817462ead3ff3e759e92b16 not found: ID does not exist" Jul 22 02:13:47.380209 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.380178 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:13:47.384000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:47.383974 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4735213f-openshift-default-5744b8bb6c-dw5wx"] Jul 22 02:13:48.678107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:48.678067 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d53179-bad2-4258-8b86-52152eeca379" path="/var/lib/kubelet/pods/26d53179-bad2-4258-8b86-52152eeca379/volumes" Jul 22 02:13:51.415289 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:51.415251 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:51.568109 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:51.568071 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:51.699929 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:51.699834 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:51.793807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:51.793772 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:13:52.242706 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:13:52.242669 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:18.353513 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:18.353470 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:25.618152 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:25.618108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:25.720688 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:25.720629 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:25.836558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:25.836516 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:33.110465 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:33.110423 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:41.938351 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:41.938289 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:47.844566 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:47.844519 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:47.914530 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:47.914483 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:48.028448 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:48.028411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:48.128570 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:48.128486 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:58.731836 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:58.731788 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:14:59.581279 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:14:59.581241 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:06.331597 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:06.331558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:06.438026 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:06.437984 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:06.538880 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:06.538847 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:06.676984 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:06.676949 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:12.700454 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:12.700411 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:12.799826 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:12.799771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:12.901103 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:12.901071 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:14.798384 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:14.798346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:20.713463 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:20.713425 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:20.816918 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:20.816871 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:21.524709 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:21.524651 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.467041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:27.466993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.569841 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:27.569794 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:27.740162 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:27.740074 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:28.042355 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.042271 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:15:28.043026 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043002 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.043232 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043207 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043244 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043253 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043283 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043289 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043295 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043300 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043714 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043418 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.043714 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043438 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043714 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043449 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecfa400-742a-4927-b0ce-3042bd68c01d" containerName="istio-proxy" Jul 22 02:15:28.043714 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.043466 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d53179-bad2-4258-8b86-52152eeca379" containerName="istio-proxy" Jul 22 02:15:28.046932 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.046901 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.049900 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.049880 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4e4fadaa-openshift-default-dockercfg-qxgvs\"" Jul 22 02:15:28.057713 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.057688 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:15:28.072447 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.072418 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:28.139348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139403 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139467 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139558 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139912 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139912 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.139912 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.139774 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx868\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.240722 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.240722 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx868\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.240979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.240979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.240979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241142 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.240983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241142 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241034 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241142 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241142 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241128 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241335 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241335 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241245 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241417 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241396 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241489 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.241526 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.241479 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.243360 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.243343 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.243488 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.243472 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.249617 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.249589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.249854 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.249833 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx868\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868\") pod \"e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.360040 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.359938 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:28.467915 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.467692 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:28.519194 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.519164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:15:28.521059 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:15:28.521030 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba80d03f_b5cb_4682_8e81_23b3bafcf938.slice/crio-f52c313cfba0a5b13dccd8eb3d359841a0adb0e34bc2d5ebb16929da602b20e7 WatchSource:0}: Error finding container f52c313cfba0a5b13dccd8eb3d359841a0adb0e34bc2d5ebb16929da602b20e7: Status 404 returned error can't find the container with id f52c313cfba0a5b13dccd8eb3d359841a0adb0e34bc2d5ebb16929da602b20e7 Jul 22 02:15:28.523227 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.523195 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:28.523327 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.523277 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:28.523327 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.523322 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:28.854937 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.854896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerStarted","Data":"385560976fc8fd1da81903c7fe5938be2f9b00cb26a9f0603d128d4a1697c6b4"} Jul 22 02:15:28.855155 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.854959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerStarted","Data":"f52c313cfba0a5b13dccd8eb3d359841a0adb0e34bc2d5ebb16929da602b20e7"} Jul 22 02:15:28.890280 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:28.890225 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podStartSLOduration=0.89020735 podStartE2EDuration="890.20735ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:28.888950843 +0000 UTC m=+1806.789829341" watchObservedRunningTime="2026-07-22 02:15:28.89020735 +0000 UTC m=+1806.791085846" Jul 22 02:15:29.360901 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:29.360852 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:29.365821 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:29.365796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:29.860091 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:29.860060 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:29.861190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:29.861165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:31.204579 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:31.204522 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:31.298702 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:31.298663 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:31.443685 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:31.443619 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:31.785105 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:31.785066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:32.073284 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:32.073179 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:32.173682 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:32.173630 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:35.762957 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:35.762916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:36.501772 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:36.501735 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:36.601994 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:36.601952 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:36.983559 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:36.983519 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:42.974309 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:42.974259 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:43.045910 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:43.045850 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:44.862218 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.862188 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 02:15:44.862611 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.862254 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 02:15:44.937466 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.937435 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerID="385560976fc8fd1da81903c7fe5938be2f9b00cb26a9f0603d128d4a1697c6b4" exitCode=0 Jul 22 02:15:44.937661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.937510 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerDied","Data":"385560976fc8fd1da81903c7fe5938be2f9b00cb26a9f0603d128d4a1697c6b4"} Jul 22 02:15:44.937956 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.937943 2585 scope.go:117] "RemoveContainer" containerID="385560976fc8fd1da81903c7fe5938be2f9b00cb26a9f0603d128d4a1697c6b4" Jul 22 02:15:44.938985 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.938957 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:44.939061 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.939043 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:44.939098 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:44.939087 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:15:45.948365 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:45.948330 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerStarted","Data":"77fe89d4d70fc20800d45f5ac7080d96e1e7807c33e367136119a2482fbb903d"} Jul 22 02:15:46.361197 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:46.361091 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:46.361359 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:46.361327 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 02:15:46.361403 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:46.361381 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 02:15:47.361014 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:47.360969 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 02:15:47.361512 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:47.361036 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 02:15:48.360614 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:48.360574 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 02:15:48.360819 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:48.360662 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 02:15:49.360590 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:49.360548 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 22 02:15:49.361030 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:49.360623 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 22 02:15:50.364951 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:50.364919 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:50.365424 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:50.365329 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:50.366062 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:50.366042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:15:53.966311 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:53.966276 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:58.444655 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:58.444575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:15:58.890241 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:15:58.890150 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:04.094120 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.094083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:04.194829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.194791 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:04.735584 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.735541 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:04.833198 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.833165 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:04.916844 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.916812 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:16:04.917059 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.917038 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" containerID="cri-o://77fe89d4d70fc20800d45f5ac7080d96e1e7807c33e367136119a2482fbb903d" gracePeriod=30 Jul 22 02:16:04.933079 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:04.933041 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:05.192282 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.192246 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:05.355789 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.355749 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:05.367787 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.367752 2585 patch_prober.go:28] interesting pod/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 02:16:05.367923 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.367847 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 02:16:05.456035 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.455934 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:05.639084 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:05.639042 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:10.079550 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.079513 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerID="77fe89d4d70fc20800d45f5ac7080d96e1e7807c33e367136119a2482fbb903d" exitCode=0 Jul 22 02:16:10.080024 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.079590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerDied","Data":"77fe89d4d70fc20800d45f5ac7080d96e1e7807c33e367136119a2482fbb903d"} Jul 22 02:16:10.080024 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.079686 2585 scope.go:117] "RemoveContainer" containerID="385560976fc8fd1da81903c7fe5938be2f9b00cb26a9f0603d128d4a1697c6b4" Jul 22 02:16:10.172613 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.172589 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:16:10.282015 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.281926 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282015 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.281970 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282015 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282014 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282269 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282062 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx868\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282269 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282112 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282269 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282159 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282269 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282182 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282269 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282224 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282464 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282297 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs\") pod \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\" (UID: \"ba80d03f-b5cb-4682-8e81-23b3bafcf938\") " Jul 22 02:16:10.282529 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282507 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:16:10.282717 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282629 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:10.282938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282733 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.282938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:10.282938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282745 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:10.282938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.282913 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:10.284580 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.284559 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:16:10.284724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.284665 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868" (OuterVolumeSpecName: "kube-api-access-hx868") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "kube-api-access-hx868". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:16:10.284869 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.284852 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:10.284936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.284881 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba80d03f-b5cb-4682-8e81-23b3bafcf938" (UID: "ba80d03f-b5cb-4682-8e81-23b3bafcf938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:16:10.383389 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383350 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383389 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383383 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx868\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-kube-api-access-hx868\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383389 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383393 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383403 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383413 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383421 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383430 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:10.383622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:10.383439 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba80d03f-b5cb-4682-8e81-23b3bafcf938-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:16:11.085979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.085899 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" Jul 22 02:16:11.085979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.085939 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7" event={"ID":"ba80d03f-b5cb-4682-8e81-23b3bafcf938","Type":"ContainerDied","Data":"f52c313cfba0a5b13dccd8eb3d359841a0adb0e34bc2d5ebb16929da602b20e7"} Jul 22 02:16:11.086449 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.085998 2585 scope.go:117] "RemoveContainer" containerID="77fe89d4d70fc20800d45f5ac7080d96e1e7807c33e367136119a2482fbb903d" Jul 22 02:16:11.105617 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.105581 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:16:11.110153 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.110125 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4e4fadaa-openshift-default-7f95f988bc-24dr7"] Jul 22 02:16:11.552573 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.552535 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:11.658646 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.658590 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:11.753158 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.753108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:11.944399 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:11.944361 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:12.679023 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:12.678973 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" path="/var/lib/kubelet/pods/ba80d03f-b5cb-4682-8e81-23b3bafcf938/volumes" Jul 22 02:16:17.190891 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:17.190855 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:17.264407 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:17.264367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:17.866823 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:17.866786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:17.926123 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:17.926093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:24.974409 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:24.974373 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:25.275668 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.275552 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:16:25.276212 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276186 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.276212 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276210 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.276350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276242 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.276350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276251 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.276350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276343 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.276459 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.276357 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba80d03f-b5cb-4682-8e81-23b3bafcf938" containerName="istio-proxy" Jul 22 02:16:25.278907 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.278885 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.281495 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.281469 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2f4841f-openshift-default-dockercfg-xstwg\"" Jul 22 02:16:25.290566 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.290540 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:16:25.300071 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.300043 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:25.329664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.329839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329673 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.329839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.329839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.329839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329800 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.329839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqxs\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.330010 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.330010 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329902 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.330010 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.329924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.421836 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.421795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:25.431335 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431303 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431335 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431573 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431573 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431573 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.431829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.432051 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqxs\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.432051 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.431910 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.432051 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.432028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.432361 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.432328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.432469 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.432409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.434062 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.434038 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.434166 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.434112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.440021 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.439997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.440138 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.440088 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqxs\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs\") pod \"e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.592628 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.592525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:25.733401 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.733374 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:16:25.735242 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:16:25.735211 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f273b3_6441_4eb1_9d2a_9c374e04bae9.slice/crio-053ab573c8aca51be01f08dba234b66607a0933ed98d79d7835ea58ba407d552 WatchSource:0}: Error finding container 053ab573c8aca51be01f08dba234b66607a0933ed98d79d7835ea58ba407d552: Status 404 returned error can't find the container with id 053ab573c8aca51be01f08dba234b66607a0933ed98d79d7835ea58ba407d552 Jul 22 02:16:25.737421 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.737386 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:25.737489 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.737471 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:25.737527 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:25.737518 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:26.161924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:26.161892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerStarted","Data":"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2"} Jul 22 02:16:26.162325 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:26.161943 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerStarted","Data":"053ab573c8aca51be01f08dba234b66607a0933ed98d79d7835ea58ba407d552"} Jul 22 02:16:26.186955 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:26.186889 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podStartSLOduration=1.186871241 podStartE2EDuration="1.186871241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:26.184001899 +0000 UTC m=+1864.084880421" watchObservedRunningTime="2026-07-22 02:16:26.186871241 +0000 UTC m=+1864.087749738" Jul 22 02:16:26.593171 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:26.593132 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:26.598125 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:26.598097 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:27.168131 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:27.168097 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:27.169166 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:27.169144 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:28.177458 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:28.177419 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:28.275839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:28.275797 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:28.383394 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:28.383322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:28.483268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:28.483164 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:28.877472 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:28.877374 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:30.025798 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:30.025750 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:32.738827 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:32.738789 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:37.239197 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:37.239161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:16:41.262865 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.262830 2585 generic.go:356] "Generic (PLEG): container finished" podID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerID="d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2" exitCode=0 Jul 22 02:16:41.263299 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.262905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerDied","Data":"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2"} Jul 22 02:16:41.263355 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.263341 2585 scope.go:117] "RemoveContainer" containerID="d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2" Jul 22 02:16:41.264324 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.264300 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:41.264383 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.264369 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:41.264420 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:41.264408 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:16:42.168358 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:42.168323 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:42.274878 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:42.274841 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerStarted","Data":"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c"} Jul 22 02:16:42.593088 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:42.592993 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:42.594231 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:42.594206 2585 patch_prober.go:28] interesting pod/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 22 02:16:42.594355 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:42.594254 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 22 02:16:43.593304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:43.593266 2585 patch_prober.go:28] interesting pod/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 22 02:16:43.593710 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:43.593321 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 22 02:16:44.593756 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:44.593717 2585 patch_prober.go:28] interesting pod/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 22 02:16:44.594245 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:44.593786 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 22 02:16:45.593048 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:45.593011 2585 patch_prober.go:28] interesting pod/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 22 02:16:45.593268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:45.593075 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 22 02:16:46.597454 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:46.597423 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:46.597891 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:46.597698 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:16:46.598529 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:16:46.598512 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:17:12.205456 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:17:12.205414 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:16.651217 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:17:16.651178 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:19.093035 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:17:19.093004 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:17:24.133135 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:17:24.133102 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:01.153476 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:01.153431 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:01.462652 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:01.462597 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:07.070844 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.070802 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:07.171240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.171201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:07.354898 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.354816 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:18:07.355115 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.355086 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" containerID="cri-o://5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c" gracePeriod=30 Jul 22 02:18:07.377486 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.377449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:07.635158 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:07.635069 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:08.719869 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:08.719828 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:08.821227 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:08.821190 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:08.918108 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:08.918070 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:09.701291 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:09.701256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:09.799921 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:09.799878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:09.900016 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:09.899976 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:11.448255 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:11.448220 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:12.622915 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.622847 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:18:12.742393 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.742357 2585 generic.go:356] "Generic (PLEG): container finished" podID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerID="5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c" exitCode=0 Jul 22 02:18:12.742589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.742406 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" Jul 22 02:18:12.742589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.742407 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerDied","Data":"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c"} Jul 22 02:18:12.742589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.742462 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47" event={"ID":"52f273b3-6441-4eb1-9d2a-9c374e04bae9","Type":"ContainerDied","Data":"053ab573c8aca51be01f08dba234b66607a0933ed98d79d7835ea58ba407d552"} Jul 22 02:18:12.742589 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.742486 2585 scope.go:117] "RemoveContainer" containerID="5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c" Jul 22 02:18:12.751585 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.751565 2585 scope.go:117] "RemoveContainer" containerID="d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2" Jul 22 02:18:12.759670 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.759627 2585 scope.go:117] "RemoveContainer" containerID="5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c" Jul 22 02:18:12.759948 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:18:12.759930 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c\": container with ID starting with 5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c not found: ID does not exist" containerID="5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c" Jul 22 02:18:12.760003 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.759959 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c"} err="failed to get container status \"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c\": rpc error: code = NotFound desc = could not find container \"5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c\": container with ID starting with 5fd672ce093db6836f80b18adc99c41a6cb3926aa38173bfb5cf66a6722dc26c not found: ID does not exist" Jul 22 02:18:12.760003 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.759980 2585 scope.go:117] "RemoveContainer" containerID="d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2" Jul 22 02:18:12.760211 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:18:12.760193 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2\": container with ID starting with d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2 not found: ID does not exist" containerID="d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2" Jul 22 02:18:12.760260 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.760216 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2"} err="failed to get container status \"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2\": rpc error: code = NotFound desc = could not find container \"d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2\": container with ID starting with d641781b40fbf508dacc7a1e1a52ad6deb72fcd0c637eadf5f4e2b6339ad81a2 not found: ID does not exist" Jul 22 02:18:12.766125 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766107 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766228 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766153 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766228 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766175 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqxs\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766228 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766212 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766294 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766329 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:18:12.766388 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766333 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766537 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766394 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766537 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766444 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766537 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766471 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs\") pod \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\" (UID: \"52f273b3-6441-4eb1-9d2a-9c374e04bae9\") " Jul 22 02:18:12.766719 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766547 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:18:12.766851 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766793 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:18:12.767149 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766903 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:18:12.767298 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.766948 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.767298 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.767191 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.767298 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.767256 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data" (OuterVolumeSpecName: "istio-data") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:18:12.768349 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.768327 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs" (OuterVolumeSpecName: "kube-api-access-4rqxs") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "kube-api-access-4rqxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:18:12.768442 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.768391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:18:12.768864 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.768849 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token" (OuterVolumeSpecName: "istio-token") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:18:12.768985 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.768914 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52f273b3-6441-4eb1-9d2a-9c374e04bae9" (UID: "52f273b3-6441-4eb1-9d2a-9c374e04bae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:18:12.868377 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868335 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868377 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868370 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqxs\" (UniqueName: \"kubernetes.io/projected/52f273b3-6441-4eb1-9d2a-9c374e04bae9-kube-api-access-4rqxs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868377 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868383 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868628 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868393 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868628 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868405 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868628 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868415 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:12.868628 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:12.868425 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f273b3-6441-4eb1-9d2a-9c374e04bae9-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:18:13.066860 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:13.066820 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:18:13.069988 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:13.069956 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2f4841f-openshift-default-788bd69b59-kqf47"] Jul 22 02:18:14.676946 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:14.676879 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" path="/var/lib/kubelet/pods/52f273b3-6441-4eb1-9d2a-9c374e04bae9/volumes" Jul 22 02:18:16.388916 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:16.388870 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:17.843620 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:17.843583 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:17.926414 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:17.926373 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:18.552724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:18.552677 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:21.916962 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:21.916911 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:22.017853 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:22.017814 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:24.439811 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:24.439771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:24.544562 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:24.544531 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:24.640417 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:24.640375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:35.189871 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:35.189830 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:35.554455 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:35.554363 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:18:56.569294 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:18:56.569252 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:02.774596 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:02.774556 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:02.940433 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:02.940395 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:03.160438 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.160349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:03.458008 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.457974 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:19:03.458408 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.458395 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:03.458453 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.458409 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:03.458491 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.458484 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:03.458529 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.458493 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:03.461673 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.461657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.464489 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.464455 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f059e-openshift-default-dockercfg-x7dhg\"" Jul 22 02:19:03.476906 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.476863 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:19:03.487875 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.487842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488060 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.487888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488060 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.487930 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488157 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488157 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488104 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488239 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488239 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488318 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnbg\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488370 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.488743 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.488717 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:03.589796 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.589796 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589917 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.589994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnbg\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590032 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590407 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590465 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590465 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590450 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590562 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590540 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.590748 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.590731 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.592174 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.592157 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.592443 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.592425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.605731 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.605700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.613453 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.613426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnbg\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg\") pod \"e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.647600 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.647558 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:03.772894 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.772778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:03.922249 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.922221 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:19:03.924844 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:19:03.924816 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3efa2aa_4339_4d58_9d70_bea1915f5e16.slice/crio-752c6d231e7ae6195cc25370f8a7f3c1216846c39f132222e2bbfc3039b446cb WatchSource:0}: Error finding container 752c6d231e7ae6195cc25370f8a7f3c1216846c39f132222e2bbfc3039b446cb: Status 404 returned error can't find the container with id 752c6d231e7ae6195cc25370f8a7f3c1216846c39f132222e2bbfc3039b446cb Jul 22 02:19:03.927115 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.927084 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:03.927178 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.927151 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:03.927221 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.927177 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:03.994321 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.994287 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerStarted","Data":"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e"} Jul 22 02:19:03.994477 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:03.994346 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerStarted","Data":"752c6d231e7ae6195cc25370f8a7f3c1216846c39f132222e2bbfc3039b446cb"} Jul 22 02:19:04.014412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:04.014353 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podStartSLOduration=1.014332938 podStartE2EDuration="1.014332938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:04.013235349 +0000 UTC m=+2021.914113846" watchObservedRunningTime="2026-07-22 02:19:04.014332938 +0000 UTC m=+2021.915211434" Jul 22 02:19:04.773586 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:04.773553 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:04.778457 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:04.778432 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:04.999755 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:04.999723 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:05.000812 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:05.000791 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:05.987275 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:05.987237 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:06.079818 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:06.079782 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:06.164408 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:06.164373 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:06.466742 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:06.466702 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:06.681028 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:06.680994 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:06.783243 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:06.783152 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:10.569682 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:10.569620 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:10.907201 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:10.907105 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:11.207629 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.207598 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:19:11.208072 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.208057 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:11.208123 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.208073 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f273b3-6441-4eb1-9d2a-9c374e04bae9" containerName="istio-proxy" Jul 22 02:19:11.211704 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.211672 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.214673 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.214648 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7e7df6-openshift-default-dockercfg-62brf\"" Jul 22 02:19:11.225910 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.225874 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:19:11.239994 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.239960 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:11.270307 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270267 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270307 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270304 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270534 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270346 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270534 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270534 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270534 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8j7\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270711 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270711 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.270711 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.270625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.371861 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371819 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.371861 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371884 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.371982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372076 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372125 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8j7\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372505 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372505 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372393 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372505 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372482 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372702 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.372810 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.372792 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.374367 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.374346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.374587 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.374568 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.389252 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.389221 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.389724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.389703 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8j7\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7\") pod \"e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.522816 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.522727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:11.528152 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.528113 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:11.668226 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.668195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:19:11.670587 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:19:11.670549 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb1cf5a_145d_4001_a0d7_b13479dda74d.slice/crio-8508a225e7ab27c74532b29c987aa1c0f853ef8215c72115bb2fcb2d04f0b05c WatchSource:0}: Error finding container 8508a225e7ab27c74532b29c987aa1c0f853ef8215c72115bb2fcb2d04f0b05c: Status 404 returned error can't find the container with id 8508a225e7ab27c74532b29c987aa1c0f853ef8215c72115bb2fcb2d04f0b05c Jul 22 02:19:11.672967 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.672932 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:11.673037 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.673017 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:11.673086 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:11.673052 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:12.044563 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:12.044523 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerStarted","Data":"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c"} Jul 22 02:19:12.044770 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:12.044608 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerStarted","Data":"8508a225e7ab27c74532b29c987aa1c0f853ef8215c72115bb2fcb2d04f0b05c"} Jul 22 02:19:12.066455 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:12.066396 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podStartSLOduration=1.06637862 podStartE2EDuration="1.06637862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:12.063994659 +0000 UTC m=+2029.964873159" watchObservedRunningTime="2026-07-22 02:19:12.06637862 +0000 UTC m=+2029.967257117" Jul 22 02:19:12.523841 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:12.523802 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:12.528845 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:12.528816 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:13.050704 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:13.050671 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:13.051839 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:13.051815 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:14.108824 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.108785 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:14.215982 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.215919 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:14.305987 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.305953 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:14.682673 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.682623 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:14.816000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.815963 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:14.918922 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:14.918878 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:18.659696 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:18.659660 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:20.000985 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.000897 2585 patch_prober.go:28] interesting pod/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 02:19:20.000985 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.000962 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 02:19:20.090793 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.090759 2585 generic.go:356] "Generic (PLEG): container finished" podID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerID="6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e" exitCode=0 Jul 22 02:19:20.090967 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.090829 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerDied","Data":"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e"} Jul 22 02:19:20.091266 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.091252 2585 scope.go:117] "RemoveContainer" containerID="6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e" Jul 22 02:19:20.092349 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.092321 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:20.092425 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.092412 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:20.092463 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:20.092445 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:21.097720 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:21.097685 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerStarted","Data":"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d"} Jul 22 02:19:21.773816 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:21.773780 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:21.774031 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:21.773998 2585 patch_prober.go:28] interesting pod/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 02:19:21.774168 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:21.774051 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 02:19:22.774215 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:22.774173 2585 patch_prober.go:28] interesting pod/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 02:19:22.774629 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:22.774244 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 02:19:23.774011 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:23.773964 2585 patch_prober.go:28] interesting pod/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 02:19:23.774231 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:23.774022 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 02:19:24.773860 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:24.773821 2585 patch_prober.go:28] interesting pod/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 02:19:24.774240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:24.773879 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 02:19:25.777728 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:25.777690 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:25.778145 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:25.777963 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:25.778650 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:25.778614 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:19:27.746733 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:27.746694 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:27.891102 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:27.891063 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:27.987151 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:27.987115 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:28.227924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:28.227885 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:29.091852 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:29.091815 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 02:19:29.092338 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:29.091992 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 02:19:31.159925 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.159832 2585 generic.go:356] "Generic (PLEG): container finished" podID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerID="454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c" exitCode=0 Jul 22 02:19:31.159925 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.159907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerDied","Data":"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c"} Jul 22 02:19:31.160367 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.160353 2585 scope.go:117] "RemoveContainer" containerID="454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c" Jul 22 02:19:31.161410 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.161386 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:31.161484 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.161451 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:31.161522 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:31.161482 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:19:32.170542 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:32.170510 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerStarted","Data":"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9"} Jul 22 02:19:32.523474 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:32.523377 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:32.523627 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:32.523578 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 22 02:19:32.523724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:32.523625 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 22 02:19:33.533167 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:33.533123 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 22 02:19:33.533830 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:33.533791 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 22 02:19:34.524119 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:34.524074 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 22 02:19:34.524343 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:34.524158 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 22 02:19:35.523488 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:35.523448 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 22 02:19:35.524121 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:35.524086 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 22 02:19:36.523557 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:36.523514 2585 patch_prober.go:28] interesting pod/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 22 02:19:36.524230 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:36.524196 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 22 02:19:36.859457 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:36.859358 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:36.957942 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:36.957902 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:19:37.527004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:37.526975 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:37.527431 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:37.527281 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:19:37.528039 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:19:37.528018 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:20:11.367119 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:20:11.367083 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:53.890390 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:20:53.890354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:54.793409 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:20:54.793367 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:55.485899 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:20:55.485862 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:20:55.576383 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:20:55.576346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:02.322140 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:02.322101 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:05.639657 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:05.639607 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:10.010985 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:10.010945 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:10.442106 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:10.442067 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:16.382496 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.382452 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:16.484540 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.484502 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:16.667148 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.667114 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:21:16.667345 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.667317 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" containerID="cri-o://bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d" gracePeriod=30 Jul 22 02:21:16.688568 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.688526 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:16.961139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:16.961021 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:17.717549 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:17.717504 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:17.847056 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:17.847017 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:21.933046 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:21.932984 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:21:22.057333 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057292 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057355 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057382 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057423 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057537 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057571 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057598 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqnbg\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057741 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.057807 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057747 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:22.057998 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057807 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:21:22.057998 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.057858 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket\") pod \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\" (UID: \"b3efa2aa-4339-4d58-9d70-bea1915f5e16\") " Jul 22 02:21:22.058117 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058097 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:22.058179 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058035 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:22.058481 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058453 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.058481 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058476 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.058631 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058490 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.058631 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058503 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.058887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.058865 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:22.059983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.059952 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg" (OuterVolumeSpecName: "kube-api-access-fqnbg") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "kube-api-access-fqnbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:22.060097 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.060015 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:22.060097 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.060069 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:21:22.060221 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.060117 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3efa2aa-4339-4d58-9d70-bea1915f5e16" (UID: "b3efa2aa-4339-4d58-9d70-bea1915f5e16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:22.159726 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.159691 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnbg\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-kube-api-access-fqnbg\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.159726 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.159722 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.159726 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.159732 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.159966 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.159740 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.159966 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.159748 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3efa2aa-4339-4d58-9d70-bea1915f5e16-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:22.752811 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.752774 2585 generic.go:356] "Generic (PLEG): container finished" podID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerID="bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d" exitCode=0 Jul 22 02:21:22.753007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.752822 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" Jul 22 02:21:22.753007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.752857 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerDied","Data":"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d"} Jul 22 02:21:22.753007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.752922 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn" event={"ID":"b3efa2aa-4339-4d58-9d70-bea1915f5e16","Type":"ContainerDied","Data":"752c6d231e7ae6195cc25370f8a7f3c1216846c39f132222e2bbfc3039b446cb"} Jul 22 02:21:22.753007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.752944 2585 scope.go:117] "RemoveContainer" containerID="bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d" Jul 22 02:21:22.762121 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.762085 2585 scope.go:117] "RemoveContainer" containerID="6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e" Jul 22 02:21:22.771230 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.771209 2585 scope.go:117] "RemoveContainer" containerID="bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d" Jul 22 02:21:22.771522 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:21:22.771500 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d\": container with ID starting with bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d not found: ID does not exist" containerID="bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d" Jul 22 02:21:22.771622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.771532 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d"} err="failed to get container status \"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d\": rpc error: code = NotFound desc = could not find container \"bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d\": container with ID starting with bdcbfe6227629e196d2134ae7a977d35a090c368653eb874afafb230b5da805d not found: ID does not exist" Jul 22 02:21:22.771622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.771551 2585 scope.go:117] "RemoveContainer" containerID="6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e" Jul 22 02:21:22.771834 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:21:22.771808 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e\": container with ID starting with 6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e not found: ID does not exist" containerID="6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e" Jul 22 02:21:22.771875 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.771839 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e"} err="failed to get container status \"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e\": rpc error: code = NotFound desc = could not find container \"6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e\": container with ID starting with 6b39e09e9b95dfaf006758e712608b5c0c1ca9291b5dfe1be452f5e5e5e0500e not found: ID does not exist" Jul 22 02:21:22.772600 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.772580 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:21:22.776099 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:22.776073 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f059e-openshift-default-78545f5f77-8cvpn"] Jul 22 02:21:24.677894 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:24.677861 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" path="/var/lib/kubelet/pods/b3efa2aa-4339-4d58-9d70-bea1915f5e16/volumes" Jul 22 02:21:27.596726 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:27.596691 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:32.214322 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:32.214287 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:33.374281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:33.374238 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:34.445622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:34.445584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:34.498748 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:34.498714 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:38.648093 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:38.648046 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:38.745020 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:38.744983 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:38.942881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:38.942843 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:21:38.943121 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:38.943097 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" containerID="cri-o://08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9" gracePeriod=30 Jul 22 02:21:38.982343 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:38.982304 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:39.222075 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.221989 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:39.586303 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.586201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:39.956354 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.956323 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:21:39.956801 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.956786 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:39.956801 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.956803 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:39.956923 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.956910 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:39.956923 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.956925 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:39.960733 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.960713 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:39.963967 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.963942 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-633172-openshift-default-dockercfg-h9klq\"" Jul 22 02:21:39.976030 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.976003 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:21:39.985622 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:39.985579 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051468 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051493 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051594 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051664 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.051936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.051798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9kp\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.152871 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.152828 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153084 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.152882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153084 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.152914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153084 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.152951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153136 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9kp\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153268 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153549 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153549 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153525 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153653 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153570 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.153879 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.153862 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.155234 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.155203 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.155619 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.155597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.167955 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.167928 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.169185 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.169163 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9kp\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp\") pod \"e2e-api-a-633172-openshift-default-dd7d565f-4w778\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.206057 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.206020 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:40.270929 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.270849 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:40.430124 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:21:40.430094 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048a355c_0aa9_48cf_966a_27ac030960de.slice/crio-aa0ddfad50e97f75d57a8d516ca39340e5f2be922e6228997fc993cd1d406497 WatchSource:0}: Error finding container aa0ddfad50e97f75d57a8d516ca39340e5f2be922e6228997fc993cd1d406497: Status 404 returned error can't find the container with id aa0ddfad50e97f75d57a8d516ca39340e5f2be922e6228997fc993cd1d406497 Jul 22 02:21:40.431344 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.431320 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:21:40.432227 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.432195 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:40.432302 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.432263 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:40.432302 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.432293 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:40.845792 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.845760 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerStarted","Data":"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018"} Jul 22 02:21:40.846003 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.845815 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerStarted","Data":"aa0ddfad50e97f75d57a8d516ca39340e5f2be922e6228997fc993cd1d406497"} Jul 22 02:21:40.870724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:40.870661 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podStartSLOduration=1.870628001 podStartE2EDuration="1.870628001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:40.868692715 +0000 UTC m=+2178.769571212" watchObservedRunningTime="2026-07-22 02:21:40.870628001 +0000 UTC m=+2178.771506497" Jul 22 02:21:41.271199 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:41.271155 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:41.272918 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:41.272892 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:21:41.273055 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:41.272942 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:21:42.275153 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:42.275120 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:42.854801 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:42.854765 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:42.855863 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:42.855841 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:42.964500 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:42.964466 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:43.066305 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:43.066267 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:43.198368 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:43.198329 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:43.299321 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:43.299285 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:43.788107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:43.788066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:43.892282 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:43.892247 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:44.205869 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.205844 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:21:44.296465 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296427 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296509 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr8j7\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296547 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296570 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296831 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296740 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296831 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296800 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296942 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296849 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296942 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296898 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.296942 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296915 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:44.297082 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296955 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket\") pod \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\" (UID: \"3cb1cf5a-145d-4001-a0d7-b13479dda74d\") " Jul 22 02:21:44.297082 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.296989 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:44.297082 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297037 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:21:44.297281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297261 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:44.297383 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297369 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.297441 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297387 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.297441 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297400 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.297441 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297413 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.297774 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.297755 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:44.298999 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.298980 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:44.299090 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.299020 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:21:44.299240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.299211 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:21:44.299240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.299222 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7" (OuterVolumeSpecName: "kube-api-access-kr8j7") pod "3cb1cf5a-145d-4001-a0d7-b13479dda74d" (UID: "3cb1cf5a-145d-4001-a0d7-b13479dda74d"). InnerVolumeSpecName "kube-api-access-kr8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:21:44.398813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.398721 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.398813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.398752 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.398813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.398761 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.398813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.398770 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb1cf5a-145d-4001-a0d7-b13479dda74d-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.398813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.398779 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8j7\" (UniqueName: \"kubernetes.io/projected/3cb1cf5a-145d-4001-a0d7-b13479dda74d-kube-api-access-kr8j7\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:21:44.867803 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.867763 2585 generic.go:356] "Generic (PLEG): container finished" podID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerID="08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9" exitCode=0 Jul 22 02:21:44.867972 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.867834 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" Jul 22 02:21:44.867972 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.867842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerDied","Data":"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9"} Jul 22 02:21:44.867972 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.867891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8" event={"ID":"3cb1cf5a-145d-4001-a0d7-b13479dda74d","Type":"ContainerDied","Data":"8508a225e7ab27c74532b29c987aa1c0f853ef8215c72115bb2fcb2d04f0b05c"} Jul 22 02:21:44.867972 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.867913 2585 scope.go:117] "RemoveContainer" containerID="08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9" Jul 22 02:21:44.877204 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.877185 2585 scope.go:117] "RemoveContainer" containerID="454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c" Jul 22 02:21:44.886012 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.885996 2585 scope.go:117] "RemoveContainer" containerID="08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9" Jul 22 02:21:44.886324 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:21:44.886298 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9\": container with ID starting with 08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9 not found: ID does not exist" containerID="08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9" Jul 22 02:21:44.886412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.886333 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9"} err="failed to get container status \"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9\": rpc error: code = NotFound desc = could not find container \"08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9\": container with ID starting with 08e4cf9417c915a43fd4fd17e2df210df9d68bd79720091613d243c271e9bba9 not found: ID does not exist" Jul 22 02:21:44.886412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.886355 2585 scope.go:117] "RemoveContainer" containerID="454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c" Jul 22 02:21:44.886671 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:21:44.886654 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c\": container with ID starting with 454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c not found: ID does not exist" containerID="454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c" Jul 22 02:21:44.886722 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.886679 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c"} err="failed to get container status \"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c\": rpc error: code = NotFound desc = could not find container \"454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c\": container with ID starting with 454fcfd0c45d36ae441ed31f4a52993d0fa0ee13948ca94ee4ff3930d95c909c not found: ID does not exist" Jul 22 02:21:44.888304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.888285 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:21:44.893131 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:44.893105 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7e7df6-openshift-default-7b5d5b78c7-44bc8"] Jul 22 02:21:46.677992 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:46.677959 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" path="/var/lib/kubelet/pods/3cb1cf5a-145d-4001-a0d7-b13479dda74d/volumes" Jul 22 02:21:47.446626 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:47.446583 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:48.030672 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.030606 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:48.301927 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.301836 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:21:48.302290 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302278 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:21:48.302350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302292 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:21:48.302350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302313 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:21:48.302350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302320 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:21:48.302350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302329 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:48.302350 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302335 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3efa2aa-4339-4d58-9d70-bea1915f5e16" containerName="istio-proxy" Jul 22 02:21:48.302527 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.302418 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:21:48.307347 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.307322 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.311731 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.311703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dad16f-openshift-default-dockercfg-d79hj\"" Jul 22 02:21:48.322788 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.322759 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:21:48.352613 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.352575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:48.446412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446219 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446412 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446397 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2l6s\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446439 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.446778 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.446676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.447271 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.447242 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:48.547656 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2l6s\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.547881 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.548218 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.548218 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.547989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.548795 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.548710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.548962 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.548942 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.549038 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.549016 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.549377 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.549332 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.549476 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.549439 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.551146 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.551127 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.551450 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.551429 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.558080 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.558021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2l6s\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.561999 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.561973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token\") pod \"e2e-api-b-dad16f-openshift-default-595548f6f-ndggk\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.618182 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.618140 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:48.802995 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.802970 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:21:48.805029 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:21:48.804999 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b6403a_348c_4938_a320_caf18c4dc7ce.slice/crio-dd0331d91c05e8fa0041c97fc58b097a58bf0dbb962ae8b53a8060df91ca1b37 WatchSource:0}: Error finding container dd0331d91c05e8fa0041c97fc58b097a58bf0dbb962ae8b53a8060df91ca1b37: Status 404 returned error can't find the container with id dd0331d91c05e8fa0041c97fc58b097a58bf0dbb962ae8b53a8060df91ca1b37 Jul 22 02:21:48.807366 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.807323 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:48.807435 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.807413 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:48.807469 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.807453 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:48.892606 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.892567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerStarted","Data":"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68"} Jul 22 02:21:48.892836 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:48.892617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerStarted","Data":"dd0331d91c05e8fa0041c97fc58b097a58bf0dbb962ae8b53a8060df91ca1b37"} Jul 22 02:21:49.618720 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:49.618682 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:49.620123 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:49.620090 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 22 02:21:49.620271 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:49.620193 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 22 02:21:50.624010 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:50.623966 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:50.654581 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:50.654524 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podStartSLOduration=2.654506582 podStartE2EDuration="2.654506582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:48.938582462 +0000 UTC m=+2186.839460960" watchObservedRunningTime="2026-07-22 02:21:50.654506582 +0000 UTC m=+2188.555385078" Jul 22 02:21:50.902766 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:50.902675 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:50.904089 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:50.904071 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:21:51.070873 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:51.070826 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:51.321293 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:51.321261 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:51.647740 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:51.647621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:51.829569 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:51.829532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:52.205614 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:52.205575 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:52.303632 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:52.303591 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:54.925018 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.924974 2585 generic.go:356] "Generic (PLEG): container finished" podID="048a355c-0aa9-48cf-966a-27ac030960de" containerID="a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018" exitCode=0 Jul 22 02:21:54.925466 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.925034 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerDied","Data":"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018"} Jul 22 02:21:54.925725 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.925705 2585 scope.go:117] "RemoveContainer" containerID="a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018" Jul 22 02:21:54.927034 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.927000 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:54.927162 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.927076 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:54.927162 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:54.927108 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:21:55.529819 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:55.529790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:21:55.932444 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:55.932412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerStarted","Data":"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb"} Jul 22 02:21:56.271878 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:56.271787 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:56.273015 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:56.272986 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:21:56.273150 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:56.273040 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:21:57.271976 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:57.271942 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:21:57.272373 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:57.272002 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:21:57.854715 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:57.854681 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:21:58.271242 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:58.271197 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:21:58.271431 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:58.271264 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:21:59.271813 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:59.271773 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:21:59.272307 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:21:59.271839 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:22:00.271191 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:00.271147 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 22 02:22:00.271380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:00.271217 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 22 02:22:01.275899 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:01.275865 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:22:01.277076 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:01.277052 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:22:04.389954 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:04.389914 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:04.490979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:04.490940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:06.380293 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:06.379916 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:49292->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 22 02:22:06.380293 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:06.380003 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:49292->10.134.0.53:15021: read: connection reset by peer" Jul 22 02:22:07.000480 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.000444 2585 generic.go:356] "Generic (PLEG): container finished" podID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerID="16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68" exitCode=0 Jul 22 02:22:07.000712 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.000518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerDied","Data":"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68"} Jul 22 02:22:07.001065 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.001048 2585 scope.go:117] "RemoveContainer" containerID="16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68" Jul 22 02:22:07.002190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.002161 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:22:07.002289 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.002244 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:22:07.002351 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:07.002294 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:22:08.007387 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:08.007351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerStarted","Data":"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452"} Jul 22 02:22:08.618472 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:08.618431 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:22:08.618680 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:08.618613 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 22 02:22:08.618733 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:08.618693 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 22 02:22:09.618526 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:09.618490 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 22 02:22:09.618963 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:09.618557 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 22 02:22:10.618567 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:10.618532 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 22 02:22:10.619070 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:10.618593 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 22 02:22:11.619045 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:11.619007 2585 patch_prober.go:28] interesting pod/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 22 02:22:11.619433 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:11.619065 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 22 02:22:12.622751 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:12.622718 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:22:12.623193 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:12.623085 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:22:12.623857 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:12.623833 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:22:13.115887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:13.115856 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:13.214981 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:13.214947 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:19.021152 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:19.021104 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:23.893923 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:23.893885 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:24.323526 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:24.323459 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:30.435825 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:30.435783 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:30.540710 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:30.540672 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:30.735485 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:30.735386 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:22:30.735677 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:30.735629 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" containerID="cri-o://ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb" gracePeriod=30 Jul 22 02:22:30.765165 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:30.765126 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:31.006709 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:31.006606 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:31.278854 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:31.278767 2585 patch_prober.go:28] interesting pod/e2e-api-a-633172-openshift-default-dd7d565f-4w778 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 02:22:31.278854 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:31.278839 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 02:22:35.991568 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:35.991542 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:22:36.152348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152256 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152310 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152347 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152381 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152469 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152493 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152522 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152535 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152570 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm9kp\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.152664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152605 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy\") pod \"048a355c-0aa9-48cf-966a-27ac030960de\" (UID: \"048a355c-0aa9-48cf-966a-27ac030960de\") " Jul 22 02:22:36.153012 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152703 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:36.153012 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.152745 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:36.153107 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153031 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:22:36.153176 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153159 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.153224 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153184 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.153224 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153199 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.153224 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153213 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048a355c-0aa9-48cf-966a-27ac030960de-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.153391 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.153361 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data" (OuterVolumeSpecName: "istio-data") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:36.154810 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.154785 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token" (OuterVolumeSpecName: "istio-token") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:36.155240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.155211 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:22:36.155240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.155226 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:22:36.155240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.155236 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp" (OuterVolumeSpecName: "kube-api-access-qm9kp") pod "048a355c-0aa9-48cf-966a-27ac030960de" (UID: "048a355c-0aa9-48cf-966a-27ac030960de"). InnerVolumeSpecName "kube-api-access-qm9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:36.158949 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.158920 2585 generic.go:356] "Generic (PLEG): container finished" podID="048a355c-0aa9-48cf-966a-27ac030960de" containerID="ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb" exitCode=0 Jul 22 02:22:36.159039 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.158959 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" Jul 22 02:22:36.159039 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.158985 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerDied","Data":"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb"} Jul 22 02:22:36.159125 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.159037 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778" event={"ID":"048a355c-0aa9-48cf-966a-27ac030960de","Type":"ContainerDied","Data":"aa0ddfad50e97f75d57a8d516ca39340e5f2be922e6228997fc993cd1d406497"} Jul 22 02:22:36.159125 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.159067 2585 scope.go:117] "RemoveContainer" containerID="ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb" Jul 22 02:22:36.176298 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.176278 2585 scope.go:117] "RemoveContainer" containerID="a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018" Jul 22 02:22:36.184624 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.184594 2585 scope.go:117] "RemoveContainer" containerID="ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb" Jul 22 02:22:36.184944 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:22:36.184924 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb\": container with ID starting with ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb not found: ID does not exist" containerID="ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb" Jul 22 02:22:36.185004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.184956 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb"} err="failed to get container status \"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb\": rpc error: code = NotFound desc = could not find container \"ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb\": container with ID starting with ca89952b2084db56281941b896789465b6ee6a06f906960e5c0f3b083fb896cb not found: ID does not exist" Jul 22 02:22:36.185004 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.184974 2585 scope.go:117] "RemoveContainer" containerID="a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018" Jul 22 02:22:36.185244 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:22:36.185225 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018\": container with ID starting with a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018 not found: ID does not exist" containerID="a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018" Jul 22 02:22:36.185299 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.185250 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018"} err="failed to get container status \"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018\": rpc error: code = NotFound desc = could not find container \"a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018\": container with ID starting with a918cf2a33221e5b0c459dd73d2ff586f9ba8c5ee4cfd668ed49daa178c8b018 not found: ID does not exist" Jul 22 02:22:36.193066 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.193042 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:22:36.199303 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.199280 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-633172-openshift-default-dd7d565f-4w778"] Jul 22 02:22:36.254882 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.254848 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.254882 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.254878 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.254882 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.254887 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048a355c-0aa9-48cf-966a-27ac030960de-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.255137 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.254896 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm9kp\" (UniqueName: \"kubernetes.io/projected/048a355c-0aa9-48cf-966a-27ac030960de-kube-api-access-qm9kp\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.255137 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.254910 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048a355c-0aa9-48cf-966a-27ac030960de-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:22:36.677717 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:36.677685 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048a355c-0aa9-48cf-966a-27ac030960de" path="/var/lib/kubelet/pods/048a355c-0aa9-48cf-966a-27ac030960de/volumes" Jul 22 02:22:46.796947 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:46.796903 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:51.238761 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:51.238723 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:51.707029 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:51.706986 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:57.677653 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:57.677603 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:57.803952 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:57.803916 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:57.999585 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:57.999480 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:22:57.999793 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:57.999764 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" containerID="cri-o://96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452" gracePeriod=30 Jul 22 02:22:58.015698 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.015664 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:58.257007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.256886 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:58.652306 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.652210 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:58.975414 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975385 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:22:58.975828 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975815 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:22:58.975828 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975829 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:22:58.975908 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975838 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:22:58.975908 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975843 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:22:58.975968 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975913 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:22:58.975968 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.975924 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb1cf5a-145d-4001-a0d7-b13479dda74d" containerName="istio-proxy" Jul 22 02:22:58.977918 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.977893 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:58.981375 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.981356 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-000830-openshift-default-dockercfg-frttp\"" Jul 22 02:22:58.995069 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:58.995039 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:22:59.007385 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.007353 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077356 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnxv\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077442 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077578 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.077659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.077617 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178694 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178719 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnxv\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.178887 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179156 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.178937 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179156 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179156 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179027 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179378 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179378 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179256 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179378 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179487 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179469 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.179775 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.179754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.181148 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.181132 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.181242 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.181225 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.187498 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.187473 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.187721 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.187700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnxv\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv\") pod \"e2e-shared-a-000830-openshift-default-f44bfd555-768f6\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.254410 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.254294 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:22:59.289011 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.288973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:22:59.430264 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.430131 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:22:59.433035 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:22:59.433008 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1190d36_0d80_4f0d_be4a_5815d0fb7f87.slice/crio-fc2c3193de9c4f95d40628b74006d8252c21980b3663dd456fbdef01b2955843 WatchSource:0}: Error finding container fc2c3193de9c4f95d40628b74006d8252c21980b3663dd456fbdef01b2955843: Status 404 returned error can't find the container with id fc2c3193de9c4f95d40628b74006d8252c21980b3663dd456fbdef01b2955843 Jul 22 02:22:59.435185 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.435152 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:22:59.435282 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.435215 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:22:59.435282 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:22:59.435241 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:00.280975 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:00.280934 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerStarted","Data":"957f55ce80e71dbc222ebda9cb2f0a9eb4a3e1b52c0d53ea92d21802ecd72a62"} Jul 22 02:23:00.281383 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:00.280998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerStarted","Data":"fc2c3193de9c4f95d40628b74006d8252c21980b3663dd456fbdef01b2955843"} Jul 22 02:23:00.289590 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:00.289562 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:00.294425 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:00.294402 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:00.319619 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:00.319566 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podStartSLOduration=2.319549191 podStartE2EDuration="2.319549191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:00.317354282 +0000 UTC m=+2258.218232780" watchObservedRunningTime="2026-07-22 02:23:00.319549191 +0000 UTC m=+2258.220427687" Jul 22 02:23:01.286683 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:01.286630 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:01.287706 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:01.287688 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:02.356568 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:02.356522 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:02.448362 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:02.448324 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:02.635287 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:02.635208 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:02.914734 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:02.914683 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:03.215391 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.215360 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:03.269252 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.269228 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:23:03.298501 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.298472 2585 generic.go:356] "Generic (PLEG): container finished" podID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerID="96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452" exitCode=0 Jul 22 02:23:03.298687 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.298523 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" Jul 22 02:23:03.298687 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.298564 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerDied","Data":"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452"} Jul 22 02:23:03.298687 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.298620 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk" event={"ID":"e7b6403a-348c-4938-a320-caf18c4dc7ce","Type":"ContainerDied","Data":"dd0331d91c05e8fa0041c97fc58b097a58bf0dbb962ae8b53a8060df91ca1b37"} Jul 22 02:23:03.298687 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.298663 2585 scope.go:117] "RemoveContainer" containerID="96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452" Jul 22 02:23:03.307594 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.307576 2585 scope.go:117] "RemoveContainer" containerID="16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68" Jul 22 02:23:03.316000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.315979 2585 scope.go:117] "RemoveContainer" containerID="96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452" Jul 22 02:23:03.316281 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:23:03.316252 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452\": container with ID starting with 96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452 not found: ID does not exist" containerID="96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452" Jul 22 02:23:03.316380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.316288 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452"} err="failed to get container status \"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452\": rpc error: code = NotFound desc = could not find container \"96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452\": container with ID starting with 96c074b4dabe319392442b1fdab2edea12fe88e18a4e4f68321984a1f0ff6452 not found: ID does not exist" Jul 22 02:23:03.316380 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.316313 2585 scope.go:117] "RemoveContainer" containerID="16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68" Jul 22 02:23:03.316565 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:23:03.316546 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68\": container with ID starting with 16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68 not found: ID does not exist" containerID="16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68" Jul 22 02:23:03.316612 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.316576 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68"} err="failed to get container status \"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68\": rpc error: code = NotFound desc = could not find container \"16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68\": container with ID starting with 16363e9d7212bdf0f870a42e3e50ac5fbd68510c44872e527dec96c5edc47f68 not found: ID does not exist" Jul 22 02:23:03.419803 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.419769 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.419859 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.419932 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.419995 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420022 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2l6s\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420070 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420068 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420104 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420162 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420169 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:23:03.420278 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420199 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo\") pod \"e7b6403a-348c-4938-a320-caf18c4dc7ce\" (UID: \"e7b6403a-348c-4938-a320-caf18c4dc7ce\") " Jul 22 02:23:03.421104 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420616 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:23:03.421104 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420687 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.421104 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.420707 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.421317 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.421110 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:23:03.421317 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.421137 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:23:03.423295 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.423259 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:23:03.424002 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.423968 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:23:03.425951 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.425611 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s" (OuterVolumeSpecName: "kube-api-access-f2l6s") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "kube-api-access-f2l6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:23:03.425951 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.425877 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7b6403a-348c-4938-a320-caf18c4dc7ce" (UID: "e7b6403a-348c-4938-a320-caf18c4dc7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:23:03.428879 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.428786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:03.521621 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521588 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521621 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521619 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2l6s\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-kube-api-access-f2l6s\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521621 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521629 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521655 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b6403a-348c-4938-a320-caf18c4dc7ce-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521663 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b6403a-348c-4938-a320-caf18c4dc7ce-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521670 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.521924 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.521678 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b6403a-348c-4938-a320-caf18c4dc7ce-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:23:03.626846 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.626810 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:23:03.631086 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:03.631059 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dad16f-openshift-default-595548f6f-ndggk"] Jul 22 02:23:04.677692 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:04.677655 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" path="/var/lib/kubelet/pods/e7b6403a-348c-4938-a320-caf18c4dc7ce/volumes" Jul 22 02:23:06.927042 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:06.927006 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:07.316118 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.316019 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:07.623366 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623292 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:23:07.623749 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623736 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623751 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623760 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623766 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623897 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623853 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623897 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623862 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b6403a-348c-4938-a320-caf18c4dc7ce" containerName="istio-proxy" Jul 22 02:23:07.623897 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.623872 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="048a355c-0aa9-48cf-966a-27ac030960de" containerName="istio-proxy" Jul 22 02:23:07.626930 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.626907 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.629968 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.629932 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-7eead4-openshift-default-dockercfg-854vr\"" Jul 22 02:23:07.640843 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.640596 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:23:07.657192 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.657161 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:07.764819 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.764776 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.764853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.764912 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.764957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765202 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.765063 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765202 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.765090 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbf9\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765202 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.765111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765202 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.765129 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.765202 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.765179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.782256 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.782193 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:07.865835 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.865783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.865852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.865901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.865936 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.865999 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866041 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbf9\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866332 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866047 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866332 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866332 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866286 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866332 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866522 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866342 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866522 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866412 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866592 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866525 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.866793 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.866774 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.868392 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.868362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.868603 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.868583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.874376 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.874310 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.874493 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.874451 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbf9\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9\") pod \"e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:07.937701 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:07.937632 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:08.083260 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.083236 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:23:08.085467 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:23:08.085427 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c60c56c_6d71_4eab_8517_5b1c262cb6fb.slice/crio-1b944f692006292238f185c9aa02837c0514de100202aa480392442e9bc0ea5c WatchSource:0}: Error finding container 1b944f692006292238f185c9aa02837c0514de100202aa480392442e9bc0ea5c: Status 404 returned error can't find the container with id 1b944f692006292238f185c9aa02837c0514de100202aa480392442e9bc0ea5c Jul 22 02:23:08.087653 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.087608 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:08.087772 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.087691 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:08.087772 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.087720 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:08.326820 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.326779 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerStarted","Data":"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443"} Jul 22 02:23:08.327024 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.326878 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerStarted","Data":"1b944f692006292238f185c9aa02837c0514de100202aa480392442e9bc0ea5c"} Jul 22 02:23:08.351304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.351249 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podStartSLOduration=1.3512317280000001 podStartE2EDuration="1.351231728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:08.349589908 +0000 UTC m=+2266.250468405" watchObservedRunningTime="2026-07-22 02:23:08.351231728 +0000 UTC m=+2266.252110295" Jul 22 02:23:08.938395 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:08.938357 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:09.946011 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:09.945975 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:10.344912 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:10.344790 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:10.346563 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:10.346528 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:10.467581 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:10.467517 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:10.633366 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:10.633263 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:10.788027 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:10.787993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:11.209509 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:11.209477 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:11.335397 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:11.335364 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:11.432457 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:11.432415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:14.919490 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:14.919454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:16.596370 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:16.596032 2585 patch_prober.go:28] interesting pod/e2e-shared-a-000830-openshift-default-f44bfd555-768f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:35448->10.134.0.54:15021: read: connection reset by peer" start-of-body= Jul 22 02:23:16.596370 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:16.596094 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:35448->10.134.0.54:15021: read: connection reset by peer" Jul 22 02:23:17.384281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.384245 2585 generic.go:356] "Generic (PLEG): container finished" podID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerID="957f55ce80e71dbc222ebda9cb2f0a9eb4a3e1b52c0d53ea92d21802ecd72a62" exitCode=0 Jul 22 02:23:17.384495 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.384300 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerDied","Data":"957f55ce80e71dbc222ebda9cb2f0a9eb4a3e1b52c0d53ea92d21802ecd72a62"} Jul 22 02:23:17.384764 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.384747 2585 scope.go:117] "RemoveContainer" containerID="957f55ce80e71dbc222ebda9cb2f0a9eb4a3e1b52c0d53ea92d21802ecd72a62" Jul 22 02:23:17.385852 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.385821 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:17.385934 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.385904 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:17.385972 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:17.385944 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:18.391991 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:18.391956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerStarted","Data":"9f555d958b8a54e771c40846ff9223317d95a801ac829dab5949e1b96645668a"} Jul 22 02:23:19.289256 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:19.289214 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:19.289496 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:19.289471 2585 patch_prober.go:28] interesting pod/e2e-shared-a-000830-openshift-default-f44bfd555-768f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 22 02:23:19.289585 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:19.289533 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 22 02:23:20.290406 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:20.290362 2585 patch_prober.go:28] interesting pod/e2e-shared-a-000830-openshift-default-f44bfd555-768f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 22 02:23:20.290831 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:20.290421 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 22 02:23:21.289829 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:21.289789 2585 patch_prober.go:28] interesting pod/e2e-shared-a-000830-openshift-default-f44bfd555-768f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 22 02:23:21.290049 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:21.289864 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 22 02:23:22.289318 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.289286 2585 patch_prober.go:28] interesting pod/e2e-shared-a-000830-openshift-default-f44bfd555-768f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 22 02:23:22.289833 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.289355 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 22 02:23:22.413846 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.413762 2585 generic.go:356] "Generic (PLEG): container finished" podID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerID="1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443" exitCode=0 Jul 22 02:23:22.414006 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.413842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerDied","Data":"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443"} Jul 22 02:23:22.414295 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.414280 2585 scope.go:117] "RemoveContainer" containerID="1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443" Jul 22 02:23:22.415363 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.415336 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:22.415435 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.415421 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:22.415511 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.415499 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 02:23:22.821750 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.821715 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:22.921869 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:22.921836 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:23.292941 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.292913 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:23.293409 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.293191 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:23.293824 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.293806 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:23:23.442666 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.434721 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerStarted","Data":"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998"} Jul 22 02:23:23.938698 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.938654 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:23.939840 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.939799 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 22 02:23:23.940000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:23.939870 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 22 02:23:24.128702 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:24.128661 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:24.245724 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:24.242368 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:24.938267 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:24.938217 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 22 02:23:24.938806 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:24.938293 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 22 02:23:25.345383 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:25.345258 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:25.939116 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:25.939067 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 22 02:23:25.939664 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:25.939142 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 22 02:23:26.753032 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:26.752694 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:26.938711 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:26.938668 2585 patch_prober.go:28] interesting pod/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 22 02:23:26.939010 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:26.938979 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 22 02:23:27.942517 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:27.942483 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:27.943523 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:27.943501 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:23:32.330936 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:32.330869 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:23:32.430631 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:23:32.430599 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:24:01.144130 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:24:01.144093 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:24:14.506854 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:24:14.506806 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:25:41.672329 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:25:41.672293 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:25:53.977426 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:25:53.977328 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:26:04.786254 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:26:04.786201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:26:05.708260 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:26:05.708216 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:26:16.814053 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:26:16.813956 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:26:40.431244 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:26:40.431201 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:26:41.593883 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:26:41.593840 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:27:15.934070 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:27:15.934035 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:27:25.350661 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:27:25.350592 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:27:26.370843 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:27:26.370795 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:27:41.837411 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:27:41.837370 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:05.540876 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:05.540825 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:23.899414 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:23.899374 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:24.001436 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:24.001403 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:25.221494 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:25.221450 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:25.341220 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:25.341176 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:33.274384 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:33.274342 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:33.376598 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:33.376556 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:28:59.502626 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:28:59.502533 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:00.400102 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:00.400066 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:19.837843 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:19.837790 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:19.952317 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:19.952277 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:29.718929 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:29.718894 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:34.282358 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:34.282323 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:34.718452 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:34.718416 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:41.005659 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.005609 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:41.079043 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.078998 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:41.288754 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.288659 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:29:41.288957 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.288916 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" containerID="cri-o://9f555d958b8a54e771c40846ff9223317d95a801ac829dab5949e1b96645668a" gracePeriod=30 Jul 22 02:29:41.313870 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.313831 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:41.554092 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:41.553987 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:42.240617 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:42.240572 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:42.340390 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:42.340346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:46.464446 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.464415 2585 generic.go:356] "Generic (PLEG): container finished" podID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerID="9f555d958b8a54e771c40846ff9223317d95a801ac829dab5949e1b96645668a" exitCode=0 Jul 22 02:29:46.464832 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.464488 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerDied","Data":"9f555d958b8a54e771c40846ff9223317d95a801ac829dab5949e1b96645668a"} Jul 22 02:29:46.464832 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.464564 2585 scope.go:117] "RemoveContainer" containerID="957f55ce80e71dbc222ebda9cb2f0a9eb4a3e1b52c0d53ea92d21802ecd72a62" Jul 22 02:29:46.546874 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.546849 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:29:46.635304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635215 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635304 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635258 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635348 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635373 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635406 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635433 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpnxv\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635520 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635455 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635817 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635543 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635817 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635590 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket\") pod \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\" (UID: \"a1190d36-0d80-4f0d-be4a-5815d0fb7f87\") " Jul 22 02:29:46.635817 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635629 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:29:46.635817 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:29:46.636019 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.635998 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:29:46.636078 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636019 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:29:46.636145 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636130 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.636203 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636152 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.636203 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636167 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.636203 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636179 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.636203 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.636175 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:29:46.637761 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.637734 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:29:46.637878 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.637765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv" (OuterVolumeSpecName: "kube-api-access-tpnxv") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "kube-api-access-tpnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:29:46.637979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.637963 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:29:46.637979 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.637969 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1190d36-0d80-4f0d-be4a-5815d0fb7f87" (UID: "a1190d36-0d80-4f0d-be4a-5815d0fb7f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:29:46.737281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.737249 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.737281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.737277 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.737281 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.737286 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.737539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.737293 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:46.737539 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:46.737302 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnxv\" (UniqueName: \"kubernetes.io/projected/a1190d36-0d80-4f0d-be4a-5815d0fb7f87-kube-api-access-tpnxv\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:29:47.472008 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:47.471976 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" Jul 22 02:29:47.472447 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:47.472012 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6" event={"ID":"a1190d36-0d80-4f0d-be4a-5815d0fb7f87","Type":"ContainerDied","Data":"fc2c3193de9c4f95d40628b74006d8252c21980b3663dd456fbdef01b2955843"} Jul 22 02:29:47.472447 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:47.472072 2585 scope.go:117] "RemoveContainer" containerID="9f555d958b8a54e771c40846ff9223317d95a801ac829dab5949e1b96645668a" Jul 22 02:29:47.493686 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:47.493656 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:29:47.500459 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:47.500428 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-000830-openshift-default-f44bfd555-768f6"] Jul 22 02:29:48.678804 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:48.678765 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" path="/var/lib/kubelet/pods/a1190d36-0d80-4f0d-be4a-5815d0fb7f87/volumes" Jul 22 02:29:52.139200 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:52.139160 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:56.759528 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:56.759489 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:29:57.195588 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:29:57.195551 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:02.955671 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:02.955613 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:03.024178 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:03.024143 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:03.220263 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:03.220182 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:30:03.220451 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:03.220424 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" containerID="cri-o://1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998" gracePeriod=30 Jul 22 02:30:03.237054 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:03.237018 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:03.518034 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:03.517935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:08.483303 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.483238 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:30:08.561338 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561297 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561545 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561399 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561545 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561430 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmbf9\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561545 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561484 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561545 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561533 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561556 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561587 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.561805 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561721 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert\") pod \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\" (UID: \"3c60c56c-6d71-4eab-8517-5b1c262cb6fb\") " Jul 22 02:30:08.562009 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561826 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:30:08.562009 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.561849 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:30:08.562134 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562040 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:30:08.562190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562137 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-credential-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.562190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562150 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-socket\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.562190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562159 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-workload-certs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.562190 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562161 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:30:08.562461 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.562434 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:30:08.563677 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.563628 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:30:08.563886 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.563863 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:30:08.564074 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.564056 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9" (OuterVolumeSpecName: "kube-api-access-jmbf9") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "kube-api-access-jmbf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:30:08.564111 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.564098 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c60c56c-6d71-4eab-8517-5b1c262cb6fb" (UID: "3c60c56c-6d71-4eab-8517-5b1c262cb6fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:30:08.582159 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.582127 2585 generic.go:356] "Generic (PLEG): container finished" podID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerID="1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998" exitCode=0 Jul 22 02:30:08.582348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.582177 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" Jul 22 02:30:08.582348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.582188 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerDied","Data":"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998"} Jul 22 02:30:08.582348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.582244 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj" event={"ID":"3c60c56c-6d71-4eab-8517-5b1c262cb6fb","Type":"ContainerDied","Data":"1b944f692006292238f185c9aa02837c0514de100202aa480392442e9bc0ea5c"} Jul 22 02:30:08.582348 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.582265 2585 scope.go:117] "RemoveContainer" containerID="1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998" Jul 22 02:30:08.592071 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.592054 2585 scope.go:117] "RemoveContainer" containerID="1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443" Jul 22 02:30:08.600140 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.600122 2585 scope.go:117] "RemoveContainer" containerID="1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998" Jul 22 02:30:08.600421 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:30:08.600401 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998\": container with ID starting with 1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998 not found: ID does not exist" containerID="1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998" Jul 22 02:30:08.600491 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.600434 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998"} err="failed to get container status \"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998\": rpc error: code = NotFound desc = could not find container \"1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998\": container with ID starting with 1cb39a3ed810a4c1fb4dd76734333ae403b135f4c1ab7f6967e3c2af74ad0998 not found: ID does not exist" Jul 22 02:30:08.600491 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.600460 2585 scope.go:117] "RemoveContainer" containerID="1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443" Jul 22 02:30:08.600742 ip-10-0-133-122 kubenswrapper[2585]: E0722 02:30:08.600716 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443\": container with ID starting with 1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443 not found: ID does not exist" containerID="1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443" Jul 22 02:30:08.600836 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.600747 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443"} err="failed to get container status \"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443\": rpc error: code = NotFound desc = could not find container \"1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443\": container with ID starting with 1a061073b6dc86d157f024c5491a54543cb06c55482008e92261a11a868ab443 not found: ID does not exist" Jul 22 02:30:08.605353 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.605322 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:30:08.610358 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.610333 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-7eead4-openshift-default-5fcf87c955-5wbgj"] Jul 22 02:30:08.663172 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663134 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbf9\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-kube-api-access-jmbf9\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.663172 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663166 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-data\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.663172 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663177 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.663172 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663185 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-envoy\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.663444 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663193 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istiod-ca-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.663444 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.663200 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c60c56c-6d71-4eab-8517-5b1c262cb6fb-istio-podinfo\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\"" Jul 22 02:30:08.677180 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:08.677143 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" path="/var/lib/kubelet/pods/3c60c56c-6d71-4eab-8517-5b1c262cb6fb/volumes" Jul 22 02:30:09.930843 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:09.930809 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-969c84445-tjqh6_39ab2936-c145-4f75-9549-1e4d47a9d5f6/manager/0.log" Jul 22 02:30:10.449838 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:10.449804 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-mptdp_7b92e3ab-dcb0-467a-8c8d-f472021b34e3/manager/0.log" Jul 22 02:30:10.456915 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:10.456896 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-mptdp_7b92e3ab-dcb0-467a-8c8d-f472021b34e3/copy-manifests/0.log" Jul 22 02:30:10.964002 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:10.963971 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-685bd66c77-vhnb5_e5685683-c0f3-44fc-b99c-a3a026ef5e2f/maas-api/0.log" Jul 22 02:30:14.213386 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:14.213353 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678d87zb_85bce386-0a6b-4408-bba8-505bacc37420/istio-proxy/0.log" Jul 22 02:30:14.353030 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:14.352994 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-lt6rh_b04e3ce6-646c-4d27-b103-77ce49aed263/discovery/0.log" Jul 22 02:30:14.726815 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:14.726772 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/1.log" Jul 22 02:30:14.986967 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:14.986936 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9mfhx_c1790ce0-5b8d-487a-9859-b6054abaf5ba/payload-processing/0.log" Jul 22 02:30:15.613066 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:15.613038 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-87k4s_60ffb6dd-f3ed-4a17-87e3-a8f2f7ed2131/main/0.log" Jul 22 02:30:15.733862 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:15.733824 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gbdkk_a585b287-a7ae-4ceb-8435-5d9a72868c38/main/0.log" Jul 22 02:30:21.887443 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:21.887406 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:21.984960 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:21.984926 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:24.674434 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:24.674392 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9xd9q_9e511869-a8c9-4c7f-8926-cff6e15d2849/global-pull-secret-syncer/0.log" Jul 22 02:30:24.825910 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:24.825872 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" Jul 22 02:30:24.975197 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:24.975117 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-122.ec2.internal_2a172972fbb39a4c6031f848e0d266ff/haproxy/0.log" Jul 22 02:30:32.632311 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.632284 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/alertmanager/0.log" Jul 22 02:30:32.659827 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.659787 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/config-reloader/0.log" Jul 22 02:30:32.694070 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.694044 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/kube-rbac-proxy-web/0.log" Jul 22 02:30:32.721463 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.721432 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/kube-rbac-proxy/0.log" Jul 22 02:30:32.746981 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.746957 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/kube-rbac-proxy-metric/0.log" Jul 22 02:30:32.779418 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.779341 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/prom-label-proxy/0.log" Jul 22 02:30:32.813782 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.813751 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d256af7e-8364-4946-8e2a-09563ea920d8/init-config-reloader/0.log" Jul 22 02:30:32.908789 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.908748 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzjv7_3cf39828-cdf6-495e-96ee-b110e01dce91/kube-state-metrics/0.log" Jul 22 02:30:32.938197 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.938168 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzjv7_3cf39828-cdf6-495e-96ee-b110e01dce91/kube-rbac-proxy-main/0.log" Jul 22 02:30:32.973785 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:32.973754 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzjv7_3cf39828-cdf6-495e-96ee-b110e01dce91/kube-rbac-proxy-self/0.log" Jul 22 02:30:33.006229 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.006205 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c9bf68595-vdlpl_8278cb5b-c1e1-4457-abff-e4ee88956ad2/metrics-server/0.log" Jul 22 02:30:33.051663 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.051560 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-lgj5l_f3418968-fff5-43e9-8c51-47e2ad6dbb91/monitoring-plugin/0.log" Jul 22 02:30:33.390027 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.389941 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhrsk_1558c0c0-4b52-4771-b01a-79c8786560c6/node-exporter/0.log" Jul 22 02:30:33.433937 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.433897 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhrsk_1558c0c0-4b52-4771-b01a-79c8786560c6/kube-rbac-proxy/0.log" Jul 22 02:30:33.468959 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.468932 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhrsk_1558c0c0-4b52-4771-b01a-79c8786560c6/init-textfile/0.log" Jul 22 02:30:33.524651 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.524612 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ck5dw_ac42c982-7b05-4ba5-a82a-b1e9a1668eb1/kube-rbac-proxy-main/0.log" Jul 22 02:30:33.591460 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.591419 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ck5dw_ac42c982-7b05-4ba5-a82a-b1e9a1668eb1/kube-rbac-proxy-self/0.log" Jul 22 02:30:33.637848 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.637813 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ck5dw_ac42c982-7b05-4ba5-a82a-b1e9a1668eb1/openshift-state-metrics/0.log" Jul 22 02:30:33.691922 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.691892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/prometheus/0.log" Jul 22 02:30:33.730938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.730902 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/config-reloader/0.log" Jul 22 02:30:33.768690 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.768666 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/thanos-sidecar/0.log" Jul 22 02:30:33.806948 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.806922 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/kube-rbac-proxy-web/0.log" Jul 22 02:30:33.848473 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.848438 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/kube-rbac-proxy/0.log" Jul 22 02:30:33.887126 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.887096 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/kube-rbac-proxy-thanos/0.log" Jul 22 02:30:33.921810 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.921783 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45600ff1-4e85-4511-a211-60a23e8f13c0/init-config-reloader/0.log" Jul 22 02:30:33.973603 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:33.973517 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rm88f_c09086b7-d3dc-49ec-b888-ecf2929a580e/prometheus-operator/0.log" Jul 22 02:30:34.016696 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.016666 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rm88f_c09086b7-d3dc-49ec-b888-ecf2929a580e/kube-rbac-proxy/0.log" Jul 22 02:30:34.103142 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.103112 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86769bcb8d-sm4q8_88428074-9b3c-4d7d-9c78-7d1a58e34fd7/telemeter-client/0.log" Jul 22 02:30:34.147389 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.147361 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86769bcb8d-sm4q8_88428074-9b3c-4d7d-9c78-7d1a58e34fd7/reload/0.log" Jul 22 02:30:34.179232 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.179208 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86769bcb8d-sm4q8_88428074-9b3c-4d7d-9c78-7d1a58e34fd7/kube-rbac-proxy/0.log" Jul 22 02:30:34.232057 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.231971 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/thanos-query/0.log" Jul 22 02:30:34.262874 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.262842 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/kube-rbac-proxy-web/0.log" Jul 22 02:30:34.291465 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.291438 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/kube-rbac-proxy/0.log" Jul 22 02:30:34.331729 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.331703 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/prom-label-proxy/0.log" Jul 22 02:30:34.365169 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.365143 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/kube-rbac-proxy-rules/0.log" Jul 22 02:30:34.402095 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.402072 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74464c75bc-9z7lk_45443b09-44f5-4998-b8dd-80265857ec20/kube-rbac-proxy-metrics/0.log" Jul 22 02:30:34.474483 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474451 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx"] Jul 22 02:30:34.474928 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474915 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.474983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474930 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.474983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474945 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.474983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474951 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.474983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474960 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.474983 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474966 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474988 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.474994 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.475059 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.475067 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1190d36-0d80-4f0d-be4a-5815d0fb7f87" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.475074 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.475139 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.475079 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c60c56c-6d71-4eab-8517-5b1c262cb6fb" containerName="istio-proxy" Jul 22 02:30:34.478334 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.478317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.482104 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.482038 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wcsf\"/\"openshift-service-ca.crt\"" Jul 22 02:30:34.483166 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.483146 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6wcsf\"/\"default-dockercfg-j2hsx\"" Jul 22 02:30:34.483284 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.483179 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wcsf\"/\"kube-root-ca.crt\"" Jul 22 02:30:34.497026 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.496988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx"] Jul 22 02:30:34.519195 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.519159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-lib-modules\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.519407 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.519212 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-podres\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.519407 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.519238 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwb9p\" (UniqueName: \"kubernetes.io/projected/ea807680-05f5-4413-a91b-30745eb64138-kube-api-access-wwb9p\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.519407 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.519389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-sys\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.519583 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.519443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-proc\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.620814 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-sys\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-proc\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-lib-modules\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-sys\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-podres\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621007 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.620978 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-proc\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621177 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.621025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwb9p\" (UniqueName: \"kubernetes.io/projected/ea807680-05f5-4413-a91b-30745eb64138-kube-api-access-wwb9p\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621177 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.621040 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-lib-modules\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.621177 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.621040 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea807680-05f5-4413-a91b-30745eb64138-podres\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.634734 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.634708 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwb9p\" (UniqueName: \"kubernetes.io/projected/ea807680-05f5-4413-a91b-30745eb64138-kube-api-access-wwb9p\") pod \"perf-node-gather-daemonset-q6lxx\" (UID: \"ea807680-05f5-4413-a91b-30745eb64138\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.788502 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.788412 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:34.928505 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:34.928474 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx"] Jul 22 02:30:34.930496 ip-10-0-133-122 kubenswrapper[2585]: W0722 02:30:34.930456 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea807680_05f5_4413_a91b_30745eb64138.slice/crio-e62d2614cb13a39d959feb767414ebdbfb57483bdfb5edecf2744bd25aa931a8 WatchSource:0}: Error finding container e62d2614cb13a39d959feb767414ebdbfb57483bdfb5edecf2744bd25aa931a8: Status 404 returned error can't find the container with id e62d2614cb13a39d959feb767414ebdbfb57483bdfb5edecf2744bd25aa931a8 Jul 22 02:30:35.725193 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:35.725158 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" event={"ID":"ea807680-05f5-4413-a91b-30745eb64138","Type":"ContainerStarted","Data":"19e2e85c772b2be52c5fcd10320560b203af78297937a035b9175d7049cbff79"} Jul 22 02:30:35.725357 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:35.725211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" event={"ID":"ea807680-05f5-4413-a91b-30745eb64138","Type":"ContainerStarted","Data":"e62d2614cb13a39d959feb767414ebdbfb57483bdfb5edecf2744bd25aa931a8"} Jul 22 02:30:35.725357 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:35.725291 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:35.746694 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:35.746614 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" podStartSLOduration=1.74659474 podStartE2EDuration="1.74659474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:35.745983297 +0000 UTC m=+2713.646861815" watchObservedRunningTime="2026-07-22 02:30:35.74659474 +0000 UTC m=+2713.647473238" Jul 22 02:30:36.645275 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:36.645249 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fdb66d479-pd6dc_72cff911-4bc1-4802-9c4b-dfe6923f6fdf/console/0.log" Jul 22 02:30:38.629028 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:38.628995 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gnwr2_184fc710-7c90-400b-b82a-3a91f60fcab2/dns/0.log" Jul 22 02:30:38.657136 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:38.657106 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gnwr2_184fc710-7c90-400b-b82a-3a91f60fcab2/kube-rbac-proxy/0.log" Jul 22 02:30:38.883436 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:38.883350 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wf4c5_2e5c11bf-7804-46fd-ac48-ca4a7c3a6f58/dns-node-resolver/0.log" Jul 22 02:30:39.626505 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:39.626475 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xt2ch_1e139f97-4a64-40c6-aae3-a928b50bf79e/node-ca/0.log" Jul 22 02:30:40.661196 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:40.661163 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678d87zb_85bce386-0a6b-4408-bba8-505bacc37420/istio-proxy/0.log" Jul 22 02:30:40.886912 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:40.886881 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-lt6rh_b04e3ce6-646c-4d27-b103-77ce49aed263/discovery/0.log" Jul 22 02:30:40.979292 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:40.979259 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/0.log" Jul 22 02:30:41.031240 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:41.031206 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/1.log" Jul 22 02:30:41.131305 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:41.131273 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9mfhx_c1790ce0-5b8d-487a-9859-b6054abaf5ba/payload-processing/0.log" Jul 22 02:30:41.740085 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:41.740060 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-q6lxx" Jul 22 02:30:42.043214 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:42.043130 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bjtkx_059d8103-d9dc-47b5-9723-af409af96632/serve-healthcheck-canary/0.log" Jul 22 02:30:42.777938 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:42.777904 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-prvz2_b9332bbf-18e3-408e-8051-b2cc2b31ac69/kube-rbac-proxy/0.log" Jul 22 02:30:42.808577 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:42.808547 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-prvz2_b9332bbf-18e3-408e-8051-b2cc2b31ac69/exporter/0.log" Jul 22 02:30:42.846379 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:42.846345 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-prvz2_b9332bbf-18e3-408e-8051-b2cc2b31ac69/extractor/0.log" Jul 22 02:30:44.904621 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:44.904587 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678d87zb_85bce386-0a6b-4408-bba8-505bacc37420/istio-proxy/0.log" Jul 22 02:30:45.122852 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:45.122818 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-lt6rh_b04e3ce6-646c-4d27-b103-77ce49aed263/discovery/0.log" Jul 22 02:30:45.171171 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:45.171139 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/0.log" Jul 22 02:30:45.221535 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:45.221499 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/1.log" Jul 22 02:30:45.280089 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:45.280053 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9mfhx_c1790ce0-5b8d-487a-9859-b6054abaf5ba/payload-processing/0.log" Jul 22 02:30:45.972000 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:45.971968 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-969c84445-tjqh6_39ab2936-c145-4f75-9549-1e4d47a9d5f6/manager/0.log" Jul 22 02:30:46.302909 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.302875 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-mptdp_7b92e3ab-dcb0-467a-8c8d-f472021b34e3/manager/0.log" Jul 22 02:30:46.324032 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.324004 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-mptdp_7b92e3ab-dcb0-467a-8c8d-f472021b34e3/copy-manifests/0.log" Jul 22 02:30:46.443427 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.443395 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678d87zb_85bce386-0a6b-4408-bba8-505bacc37420/istio-proxy/0.log" Jul 22 02:30:46.658344 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.658258 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-lt6rh_b04e3ce6-646c-4d27-b103-77ce49aed263/discovery/0.log" Jul 22 02:30:46.712758 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.712724 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/0.log" Jul 22 02:30:46.768850 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.768817 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-bkzb6_be8c0946-edc7-4881-a180-233176c150c4/istio-proxy/1.log" Jul 22 02:30:46.830530 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:46.830499 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9mfhx_c1790ce0-5b8d-487a-9859-b6054abaf5ba/payload-processing/0.log" Jul 22 02:30:47.614939 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:47.614904 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7fb968c57-9dwjr_80e8127a-9d88-4dc2-bf81-279ff889c680/manager/0.log" Jul 22 02:30:50.619034 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:50.619001 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024 Jul 22 02:30:50.679548 ip-10-0-133-122 kubenswrapper[2585]: I0722 02:30:50.679508 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ftx5g_e679e408-b43d-4ba2-9400-1dd133149d09/konnectivity-agent/0.log" limit=1024