Jul 28 10:23:22.768998 ip-10-0-130-55 systemd[1]: Starting Kubernetes Kubelet... Jul 28 10:23:23.089945 ip-10-0-130-55 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:23:23.089945 ip-10-0-130-55 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 10:23:23.089945 ip-10-0-130-55 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:23:23.089945 ip-10-0-130-55 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 10:23:23.089945 ip-10-0-130-55 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:23:23.091650 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.091567 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 10:23:23.096844 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096823 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.096844 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096841 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.096844 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096845 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.096844 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096848 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096851 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096855 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096857 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096861 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096864 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096867 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096869 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096872 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096874 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096877 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096881 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096885 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096888 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096891 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096894 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096896 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096898 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096901 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.097012 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096903 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096906 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096908 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096911 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096913 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096916 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096918 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096921 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096923 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096926 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096935 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096937 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096940 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096943 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096945 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096948 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096951 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096954 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096957 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.097464 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096959 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096961 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096964 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096967 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096969 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096971 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096974 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096976 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096979 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096981 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096984 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096986 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096989 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096993 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.096997 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097000 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097003 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097006 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097009 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.097951 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097011 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097014 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097017 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097020 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097025 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097028 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097031 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097033 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097037 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097041 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097043 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097046 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097049 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097052 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097054 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097057 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097059 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097061 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097064 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097066 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.098410 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097068 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097071 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097073 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097077 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097079 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097082 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097084 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097086 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097089 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097091 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097094 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097096 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097099 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097101 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097104 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097106 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097109 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097111 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097114 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097811 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097818 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.098914 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097823 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097826 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097829 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097831 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097834 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097837 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097839 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097842 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097845 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097847 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097850 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097852 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097855 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097858 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097860 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097864 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097866 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097868 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097871 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097873 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.099421 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097876 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097878 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097881 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097883 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097885 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097888 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097890 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097893 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097895 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097898 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097900 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097903 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097906 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097908 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097911 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097913 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097916 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097918 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097920 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097923 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.099964 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097925 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097928 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097931 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097933 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097935 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097938 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097940 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097942 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097954 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097957 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097960 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097962 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097965 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097967 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097970 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097972 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097975 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097977 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097979 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097982 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.100451 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097985 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097987 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097989 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097992 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.097998 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098002 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098005 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098008 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098012 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098014 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098017 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098020 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098023 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098025 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098028 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098030 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098033 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098038 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098041 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.100933 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098043 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098046 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098050 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098054 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098058 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098061 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098063 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098066 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098069 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098072 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098074 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098077 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098079 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098082 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098084 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098087 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098089 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098094 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098452 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.101432 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098456 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098460 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098462 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098465 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098467 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098470 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098472 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098475 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098478 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098480 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098482 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098485 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098487 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098491 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098495 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098501 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098503 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098506 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098509 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.101919 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098511 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098514 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098516 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098518 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098521 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098523 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098526 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098528 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098530 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098533 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098536 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098540 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098542 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098545 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098547 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098549 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098552 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098554 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098557 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098560 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.102385 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098562 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098564 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098567 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098569 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098572 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098574 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098576 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098579 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098581 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098585 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098587 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098590 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098593 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098595 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098598 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098600 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098602 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098605 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098607 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098610 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.102885 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098612 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098615 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098617 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098622 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098625 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098628 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098631 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098634 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098636 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098638 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098641 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098643 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098645 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098648 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098650 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098653 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098657 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098659 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098662 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098664 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.103361 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098667 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098669 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098672 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098675 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098677 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098681 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098684 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098686 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098689 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098691 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098694 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098696 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098698 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098700 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098703 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098706 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098709 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098711 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.098713 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098806 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098813 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 10:23:23.103853 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098820 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098824 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098828 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098831 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098835 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098839 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098842 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098845 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098848 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098851 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098854 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098857 2567 flags.go:64] FLAG: --cgroup-root="" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098860 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098863 2567 flags.go:64] FLAG: --client-ca-file="" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098866 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098868 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098873 2567 flags.go:64] FLAG: --cluster-domain="" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098876 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098878 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098882 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098885 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098888 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098891 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 10:23:23.104363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098894 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098897 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098900 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098905 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098907 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098910 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098913 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098919 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098922 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098925 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098928 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098931 2567 flags.go:64] FLAG: --enable-server="true" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098934 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098938 2567 flags.go:64] FLAG: --event-burst="100" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098941 2567 flags.go:64] FLAG: --event-qps="50" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098943 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098959 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098962 2567 flags.go:64] FLAG: --eviction-hard="" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098967 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098970 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098973 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098976 2567 flags.go:64] FLAG: --eviction-soft="" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098979 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098982 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098985 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 10:23:23.104970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098988 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098991 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098993 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098996 2567 flags.go:64] FLAG: --feature-gates="" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.098999 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099002 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099005 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099007 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099010 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099013 2567 flags.go:64] FLAG: --help="false" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099017 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099019 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099022 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099025 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099030 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099033 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099035 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099038 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099041 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099043 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099046 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099049 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099051 2567 flags.go:64] FLAG: --kube-reserved="" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099054 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 10:23:23.105647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099056 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099059 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099062 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099064 2567 flags.go:64] FLAG: --lock-file="" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099067 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099069 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099072 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099077 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099080 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099083 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099085 2567 flags.go:64] FLAG: --logging-format="text" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099088 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099091 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099094 2567 flags.go:64] FLAG: --manifest-url="" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099096 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099100 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099103 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099106 2567 flags.go:64] FLAG: --max-pods="110" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099109 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099112 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099115 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099118 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099123 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099125 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099128 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 10:23:23.106264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099135 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099138 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099141 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099144 2567 flags.go:64] FLAG: --pod-cidr="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099146 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099151 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099154 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099157 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099159 2567 flags.go:64] FLAG: --port="10250" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099163 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099165 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce626f47533e4095" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099168 2567 flags.go:64] FLAG: --qos-reserved="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099171 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099174 2567 flags.go:64] FLAG: --register-node="true" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099176 2567 flags.go:64] FLAG: --register-with-taints="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099179 2567 flags.go:64] FLAG: --registry-burst="10" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099183 2567 flags.go:64] FLAG: --registry-qps="5" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099185 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099188 2567 flags.go:64] FLAG: --reserved-memory="" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099191 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099194 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099197 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099199 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099202 2567 flags.go:64] FLAG: --runonce="false" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099205 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 10:23:23.106901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099207 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099210 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099213 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099216 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099220 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099223 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099226 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099228 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099231 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099233 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099236 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099239 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099242 2567 flags.go:64] FLAG: --system-cgroups="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099244 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099249 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099252 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099254 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099258 2567 flags.go:64] FLAG: --tls-min-version="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099261 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099264 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099266 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099269 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099271 2567 flags.go:64] FLAG: --v="2" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099275 2567 flags.go:64] FLAG: --version="false" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099279 2567 flags.go:64] FLAG: --vmodule="" Jul 28 10:23:23.107508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099282 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.099285 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099394 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099398 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099402 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099406 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099409 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099411 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099415 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099418 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099421 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099425 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099428 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099431 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099434 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099437 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099439 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099442 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099444 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.108131 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099447 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099449 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099452 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099454 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099457 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099459 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099462 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099464 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099466 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099469 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099472 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099474 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099477 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099481 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099484 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099487 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099490 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099492 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099495 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.108615 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099498 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099500 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099504 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099507 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099509 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099513 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099516 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099518 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099521 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099523 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099525 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099528 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099530 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099533 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099536 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099538 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099540 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099543 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099545 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099548 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.109087 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099550 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099553 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099555 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099557 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099560 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099562 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099565 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099568 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099570 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099572 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099575 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099577 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099580 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099582 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099586 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099588 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099591 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099594 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099596 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099599 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.109567 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099601 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099604 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099606 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099609 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099611 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099614 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099616 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099618 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099621 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099623 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099626 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099628 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099631 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099633 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099635 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099638 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099640 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099642 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099645 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099647 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.110144 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099650 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099653 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.099655 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.100177 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.106977 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.106992 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.107627 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.107634 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107688 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107694 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107697 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107701 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107704 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107706 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107709 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.110631 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107712 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107714 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107717 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107719 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107722 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107724 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107727 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107730 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107751 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107756 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107761 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107764 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107767 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107769 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107772 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107775 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107778 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107781 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107784 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107786 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.111067 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107789 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107792 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107795 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107797 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107800 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107802 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107805 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107807 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107810 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107812 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107817 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107820 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107823 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107826 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107829 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107832 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107835 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107837 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107840 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.111546 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107842 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107845 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107847 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107850 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107852 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107854 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107857 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107859 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107862 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107864 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107867 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107870 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107872 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107876 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107880 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107883 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107886 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107889 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107892 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107894 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.112149 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107897 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107900 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107903 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107905 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107908 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107911 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107913 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107915 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107918 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107920 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107922 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107925 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107929 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107933 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107935 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107938 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107940 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107942 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107945 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.112685 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107948 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107950 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107953 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107955 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107958 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107961 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107964 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107967 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107969 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107972 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107974 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107977 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107979 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.107982 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.107987 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:23:23.113161 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108103 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108107 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108110 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108113 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108116 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108118 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108121 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108123 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108126 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108128 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108131 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108134 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108136 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108139 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108141 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108144 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108146 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108149 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108151 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:23:23.113617 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108154 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108156 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108158 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108161 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108164 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108167 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108170 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108172 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108174 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108177 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108180 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108182 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108184 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108187 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108189 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108191 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108194 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108196 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108198 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108201 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:23:23.114480 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108203 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108206 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108208 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108211 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108213 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108215 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108218 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108222 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108225 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108228 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108230 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108233 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108235 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108238 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108241 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108244 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108246 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108250 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108252 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108255 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:23:23.115229 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108257 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108260 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108262 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108266 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108269 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108272 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108275 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108277 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108279 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108282 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108284 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108287 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108289 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108291 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108294 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108296 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108298 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108301 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108303 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108306 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:23:23.115719 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108308 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108310 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108313 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108315 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108318 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108320 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108323 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108325 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108328 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108330 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108333 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108335 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108339 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108342 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108345 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108347 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108350 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108352 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108354 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:23:23.116436 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:23.108357 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.108362 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.108461 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.110504 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.111610 2567 server.go:1019] "Starting client certificate rotation" Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.111718 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.111773 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 10:23:23.116923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.114118 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 10:23:23.118299 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.118275 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 10:23:23.127530 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.127506 2567 log.go:25] "Validated CRI v1 runtime API" Jul 28 10:23:23.130393 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.130378 2567 log.go:25] "Validated CRI v1 image API" Jul 28 10:23:23.131806 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.131788 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 10:23:23.133855 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.133832 2567 fs.go:135] Filesystem UUIDs: map[29e23e09-c370-446a-8f07-bf34a4257259:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bde9448c-a537-4095-abaf-1ddcf75cda69:/dev/nvme0n1p4] Jul 28 10:23:23.133941 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.133854 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 10:23:23.138881 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.138768 2567 manager.go:216] Machine: {Timestamp:2026-07-28 10:23:23.137500051 +0000 UTC m=+0.270159322 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100604 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea3a138c38672cc69bf886c64aaea SystemUUID:ec2ea3a1-38c3-8672-cc69-bf886c64aaea BootID:ff9843dd-d727-4652-b78c-25b8b9db7005 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:d4:ae:ff:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:d4:ae:ff:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:a3:3d:d9:93:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 10:23:23.138881 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.138870 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 10:23:23.139046 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.138968 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 10:23:23.139896 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.139872 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 10:23:23.140022 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.139898 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 10:23:23.141065 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.141054 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 10:23:23.141065 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.141067 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 10:23:23.141148 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.141074 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 10:23:23.141669 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.141660 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 10:23:23.142031 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.142014 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 10:23:23.142454 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.142442 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 10:23:23.144164 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144152 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 28 10:23:23.144320 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144310 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 10:23:23.144349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144340 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 28 10:23:23.144385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144351 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 10:23:23.144385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144363 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 10:23:23.144385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144371 2567 kubelet.go:398] "Adding apiserver pod source" Jul 28 10:23:23.144385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144380 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 10:23:23.144778 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144765 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 10:23:23.144842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.144789 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 10:23:23.146616 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.146599 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 10:23:23.148451 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.148439 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 10:23:23.148491 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.148468 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 10:23:23.150074 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150053 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 10:23:23.150074 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150070 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 10:23:23.150074 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150076 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150081 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150091 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150096 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150104 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150109 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150114 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150120 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150133 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 10:23:23.150216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150138 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 10:23:23.150786 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150771 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 10:23:23.150836 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.150788 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 10:23:23.153387 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.153368 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 10:23:23.153799 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.153771 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 10:23:23.153885 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.153778 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 10:23:23.154457 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.154442 2567 server.go:1268] "Started kubelet" Jul 28 10:23:23.154594 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.154529 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 10:23:23.154808 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.154497 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 10:23:23.154808 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.154800 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 10:23:23.155470 ip-10-0-130-55 systemd[1]: Started Kubernetes Kubelet. Jul 28 10:23:23.156602 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.156482 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 10:23:23.156727 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.156711 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 28 10:23:23.160397 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.160376 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 10:23:23.162862 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.162103 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c97e55e50 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.154406992 +0000 UTC m=+0.287066265,LastTimestamp:2026-07-28 10:23:23.154406992 +0000 UTC m=+0.287066265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.165826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.165809 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 10:23:23.165897 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.165827 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 10:23:23.166044 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.166006 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.166633 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166581 2567 factory.go:55] Registering systemd factory Jul 28 10:23:23.166633 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166598 2567 factory.go:223] Registration of the systemd container factory successfully Jul 28 10:23:23.166633 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166611 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 10:23:23.166826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166640 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 10:23:23.166826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166658 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 10:23:23.166826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166802 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 28 10:23:23.166826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166810 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 28 10:23:23.166990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166902 2567 factory.go:153] Registering CRI-O factory Jul 28 10:23:23.166990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166913 2567 factory.go:223] Registration of the crio container factory successfully Jul 28 10:23:23.166990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166959 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 10:23:23.166990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166977 2567 factory.go:103] Registering Raw factory Jul 28 10:23:23.166990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.166988 2567 manager.go:1196] Started watching for new ooms in manager Jul 28 10:23:23.168043 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.168015 2567 manager.go:349] Starting recovery of all containers Jul 28 10:23:23.174574 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.174535 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 10:23:23.174574 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.174554 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 10:23:23.174916 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.174874 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 10:23:23.180407 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.180391 2567 manager.go:354] Recovery completed Jul 28 10:23:23.184348 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.184335 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.186364 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186344 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.186424 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186384 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.186424 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186395 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.186922 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186904 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 10:23:23.186922 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186921 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 10:23:23.187061 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186939 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 28 10:23:23.187061 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.186913 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.187656 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.187641 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.187713 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.187674 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.187713 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.187689 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.188571 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.188558 2567 policy_none.go:49] "None policy: Start" Jul 28 10:23:23.188627 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.188577 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 10:23:23.188627 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.188589 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 10:23:23.189555 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.189542 2567 policy_none.go:47] "Start" Jul 28 10:23:23.197454 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.197289 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd08b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186366642 +0000 UTC m=+0.319025912,LastTimestamp:2026-07-28 10:23:23.186366642 +0000 UTC m=+0.319025912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.206027 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.205958 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd62d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186389715 +0000 UTC m=+0.319048985,LastTimestamp:2026-07-28 10:23:23.186389715 +0000 UTC m=+0.319048985,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.213914 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.213833 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd865f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186398815 +0000 UTC m=+0.319058085,LastTimestamp:2026-07-28 10:23:23.186398815 +0000 UTC m=+0.319058085,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.222990 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.222929 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-55.ec2.internal.18c66d0c99cd08b2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd08b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186366642 +0000 UTC m=+0.319025912,LastTimestamp:2026-07-28 10:23:23.18765584 +0000 UTC m=+0.320315112,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.226274 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226262 2567 manager.go:337] "Starting Device Plugin manager" Jul 28 10:23:23.226338 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.226291 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 10:23:23.226338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226300 2567 server.go:87] "Starting device plugin registration server" Jul 28 10:23:23.226523 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226511 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 10:23:23.226558 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226527 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 10:23:23.226639 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226617 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 10:23:23.226706 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226695 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 10:23:23.226787 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.226707 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 10:23:23.227205 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.227185 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 10:23:23.227271 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.227230 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.231361 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.231306 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-55.ec2.internal.18c66d0c99cd62d3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd62d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186389715 +0000 UTC m=+0.319048985,LastTimestamp:2026-07-28 10:23:23.187680892 +0000 UTC m=+0.320340162,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.232108 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.232093 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-28kll" Jul 28 10:23:23.241108 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.241091 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-28kll" Jul 28 10:23:23.242824 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.242769 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-55.ec2.internal.18c66d0c99cd865f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-55.ec2.internal.18c66d0c99cd865f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-55.ec2.internal,UID:ip-10-0-130-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-55.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-55.ec2.internal,},FirstTimestamp:2026-07-28 10:23:23.186398815 +0000 UTC m=+0.319058085,LastTimestamp:2026-07-28 10:23:23.187695013 +0000 UTC m=+0.320354285,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-55.ec2.internal,}" Jul 28 10:23:23.326967 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.326943 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.329040 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.327848 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.329040 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.327878 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.329040 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.327888 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.329040 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.327920 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.336194 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.336180 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.336242 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.336201 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-55.ec2.internal\": node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.350245 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.350204 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.380222 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.380200 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 10:23:23.380293 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.380235 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 10:23:23.380293 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.380254 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 10:23:23.380372 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.380281 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 10:23:23.382333 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.382311 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:23.451330 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.451310 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.480801 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.480780 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal"] Jul 28 10:23:23.480858 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.480840 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.481712 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.481698 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.481783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.481723 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.481783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.481748 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.483950 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.483939 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.484058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.484096 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484071 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.484876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484858 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.484970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484882 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.484970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484892 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.484970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484917 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.484970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484938 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.484970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.484950 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.487178 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.487161 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.487248 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.487196 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:23:23.487853 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.487836 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:23:23.487926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.487867 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:23:23.487926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.487881 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:23:23.516377 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.516353 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-55.ec2.internal\" not found" node="ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.520730 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.520716 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-55.ec2.internal\" not found" node="ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.551672 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.551651 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.569530 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.569504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.569606 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.569533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce85c6809f8a3cdc882d4b65d5c5ac9d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"ce85c6809f8a3cdc882d4b65d5c5ac9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.569606 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.569551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.652635 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.652549 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.669918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.669896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.670003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.669923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce85c6809f8a3cdc882d4b65d5c5ac9d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"ce85c6809f8a3cdc882d4b65d5c5ac9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.670003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.669939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.670003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.669976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.670003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.669993 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b667bb6865c4baa8a162abed9eb68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal\" (UID: \"6a5b667bb6865c4baa8a162abed9eb68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.670143 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.670014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce85c6809f8a3cdc882d4b65d5c5ac9d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-55.ec2.internal\" (UID: \"ce85c6809f8a3cdc882d4b65d5c5ac9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.753117 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.753078 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.819665 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.819641 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.824600 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.824410 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:23.853857 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.853831 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.954368 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:23.954305 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:23.975720 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:23.975691 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:24.054467 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.054438 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-55.ec2.internal\" not found" Jul 28 10:23:24.090114 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.090091 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:24.111183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.111160 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 10:23:24.111316 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.111296 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:23:24.111366 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.111330 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:23:24.111366 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.111330 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:23:24.145475 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.145453 2567 apiserver.go:52] "Watching apiserver" Jul 28 10:23:24.162182 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.162149 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 10:23:24.162504 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.162482 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-j99fk","openshift-network-diagnostics/network-check-target-xnrgz","openshift-ovn-kubernetes/ovnkube-node-n5gv9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz","openshift-image-registry/node-ca-4qh9r","openshift-multus/network-metrics-daemon-4st7m","openshift-network-operator/iptables-alerter-6lsjj","kube-system/global-pull-secret-syncer-swstv","kube-system/konnectivity-agent-6p989","openshift-cluster-node-tuning-operator/tuned-vxglw","openshift-dns/node-resolver-g9dld","openshift-multus/multus-additional-cni-plugins-vwfwh"] Jul 28 10:23:24.165002 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.164984 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.165935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.165914 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 10:23:24.165935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.165933 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" Jul 28 10:23:24.167059 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167043 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:24.167214 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.167105 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:24.167520 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167504 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.167520 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167512 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 10:23:24.167625 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167527 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wmkp9\"" Jul 28 10:23:24.167625 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167571 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 10:23:24.167714 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.167624 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.169325 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.169308 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.171522 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.171505 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.173088 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173069 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173090 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173123 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-k8s-cni-cncf-io\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173132 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-netns\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173168 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-bin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173190 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2pz\" (UniqueName: \"kubernetes.io/projected/2d9a54f1-ccda-48b8-8a98-846d4781f112-kube-api-access-2s2pz\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-system-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-socket-dir-parent\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-multus\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173295 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-cnibin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-daemon-config\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-cni-binary-copy\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-kubelet\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173507 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173517 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnts6\"" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173518 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173565 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-hostroot\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-conf-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-multus-certs\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173700 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-etc-kubernetes\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.173935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-os-release\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.174297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.173966 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.174385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.174369 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 10:23:24.174444 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.174430 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.174509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.174491 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.174559 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.174506 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9j7gw\"" Jul 28 10:23:24.176100 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.176085 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.176849 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.176137 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:24.177048 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177025 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 10:23:24.177373 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177290 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" Jul 28 10:23:24.177373 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177301 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 10:23:24.177373 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177304 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.178804 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177707 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.178804 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.177831 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsrk6\"" Jul 28 10:23:24.180362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.180349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.182679 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.182658 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.182787 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.182765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.182899 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.182861 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 10:23:24.182899 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.182863 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:24.183632 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.183107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5vlj\"" Jul 28 10:23:24.183632 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.183114 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.185048 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.185032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.187387 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.187370 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.188042 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.188027 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 10:23:24.188155 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.188141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f5hhx\"" Jul 28 10:23:24.188565 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.188549 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 10:23:24.188605 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.188583 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 10:23:24.189592 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.189581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.191780 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.191761 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal"] Jul 28 10:23:24.191835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.191810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.192339 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.192323 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hp846\"" Jul 28 10:23:24.192384 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.192354 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.192609 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.192598 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s47n8\"" Jul 28 10:23:24.192908 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.192896 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.192970 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.192900 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:23:24.193447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.193435 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 10:23:24.193639 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.193613 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 10:23:24.193720 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.193708 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal"] Jul 28 10:23:24.194063 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.194051 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwtdh\"" Jul 28 10:23:24.194317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.194306 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 10:23:24.196306 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.196292 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 10:23:24.207093 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.207055 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7xv6" Jul 28 10:23:24.213937 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.213918 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7xv6" Jul 28 10:23:24.245268 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.245231 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:18:23 +0000 UTC" deadline="2028-05-05 11:06:25.412903302 +0000 UTC" Jul 28 10:23:24.245268 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.245267 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15528h43m1.167638894s" Jul 28 10:23:24.260495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.260476 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:24.267518 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.267496 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 10:23:24.274000 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.273980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-var-lib-kubelet\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-systemd\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-ovn\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274040 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbd52\" (UniqueName: \"kubernetes.io/projected/34ce0d84-c5c9-49b2-8a0f-9384349b4986-kube-api-access-tbd52\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-modprobe-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysconfig\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-netns\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-registration-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rkx\" (UniqueName: \"kubernetes.io/projected/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-kube-api-access-g4rkx\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jjh\" (UniqueName: \"kubernetes.io/projected/feeb9bdc-2726-467c-a7e8-b378efb61762-kube-api-access-p2jjh\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274252 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-konnectivity-ca\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e71134f-cb26-40d1-90e4-50a4a79e40e1-serviceca\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.274351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-kubelet-config\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-cni-binary-copy\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-etc-kubernetes\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274438 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-env-overrides\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/feeb9bdc-2726-467c-a7e8-b378efb61762-iptables-alerter-script\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274483 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzntr\" (UniqueName: \"kubernetes.io/projected/3ee4fffa-2034-4aad-9eca-4eeb515321c7-kube-api-access-rzntr\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-os-release\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-systemd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-etc-kubernetes\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-node-log\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-script-lib\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feeb9bdc-2726-467c-a7e8-b378efb61762-host-slash\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274624 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-os-release\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274627 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e71134f-cb26-40d1-90e4-50a4a79e40e1-host\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-conf\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-tuned\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.274704 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-log-socket\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-netd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-device-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-dbus\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591f7c33-2794-4cba-bc61-4b4ec073028a-tmp-dir\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274807 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pzx\" (UniqueName: \"kubernetes.io/projected/591f7c33-2794-4cba-bc61-4b4ec073028a-kube-api-access-x9pzx\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-os-release\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-cni-binary-copy\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-binary-copy\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-cnibin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-cnibin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.274973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-kubernetes\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.275358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-kubelet\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-hostroot\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-kubelet\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-conf-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-agent-certs\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-hostroot\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275187 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-conf-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-k8s-cni-cncf-io\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-netns\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-bin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275270 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-k8s-cni-cncf-io\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-netns\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-kubelet\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-bin\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm9c\" (UniqueName: \"kubernetes.io/projected/87d262f4-0010-4194-aa8d-96d95f1098bb-kube-api-access-hhm9c\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.276058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275344 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphsr\" (UniqueName: \"kubernetes.io/projected/1e71134f-cb26-40d1-90e4-50a4a79e40e1-kube-api-access-xphsr\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-run\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-lib-modules\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-socket-dir-parent\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275459 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-socket-dir-parent\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-etc-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-bin\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275529 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-sys-fs\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-system-cni-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-cnibin\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-daemon-config\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275622 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-systemd-units\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-socket-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275665 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/591f7c33-2794-4cba-bc61-4b4ec073028a-hosts-file\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-multus-certs\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.276591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2pz\" (UniqueName: \"kubernetes.io/projected/2d9a54f1-ccda-48b8-8a98-846d4781f112-kube-api-access-2s2pz\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-tmp\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275791 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-run-multus-certs\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275845 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82g2\" (UniqueName: \"kubernetes.io/projected/08a52778-2bd8-4b7b-92c6-003f40ae714b-kube-api-access-g82g2\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-host\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-system-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-multus\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-config\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.275998 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-host-var-lib-cni-multus\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-sys\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d9a54f1-ccda-48b8-8a98-846d4781f112-system-cni-dir\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276032 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-slash\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276050 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovn-node-metrics-cert\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-var-lib-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.277072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.276538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d9a54f1-ccda-48b8-8a98-846d4781f112-multus-daemon-config\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.281506 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.281487 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:24.281506 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.281505 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:24.281645 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.281514 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:24.281645 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.281579 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:24.781560789 +0000 UTC m=+1.914220061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:24.283066 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.283039 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 10:23:24.286047 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.286031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2pz\" (UniqueName: \"kubernetes.io/projected/2d9a54f1-ccda-48b8-8a98-846d4781f112-kube-api-access-2s2pz\") pod \"multus-j99fk\" (UID: \"2d9a54f1-ccda-48b8-8a98-846d4781f112\") " pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.377029 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.377029 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-sys\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-slash\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-slash\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377120 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-sys\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.377136 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovn-node-metrics-cert\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-var-lib-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.377195 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:24.877181094 +0000 UTC m=+2.009840351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.377230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-var-lib-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-var-lib-kubelet\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-systemd\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-ovn\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.377332 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-var-lib-kubelet\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377347 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-ovn\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-systemd\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbd52\" (UniqueName: \"kubernetes.io/projected/34ce0d84-c5c9-49b2-8a0f-9384349b4986-kube-api-access-tbd52\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.377390 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:24.877373631 +0000 UTC m=+2.010032893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-modprobe-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysconfig\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377483 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-netns\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-run-netns\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-registration-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.377608 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysconfig\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-registration-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377565 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rkx\" (UniqueName: \"kubernetes.io/projected/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-kube-api-access-g4rkx\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jjh\" (UniqueName: \"kubernetes.io/projected/feeb9bdc-2726-467c-a7e8-b378efb61762-kube-api-access-p2jjh\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-konnectivity-ca\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e71134f-cb26-40d1-90e4-50a4a79e40e1-serviceca\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-modprobe-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-kubelet-config\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-env-overrides\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/feeb9bdc-2726-467c-a7e8-b378efb61762-iptables-alerter-script\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzntr\" (UniqueName: \"kubernetes.io/projected/3ee4fffa-2034-4aad-9eca-4eeb515321c7-kube-api-access-rzntr\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377828 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-d\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-kubelet-config\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-systemd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.377988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-node-log\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-script-lib\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.378476 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feeb9bdc-2726-467c-a7e8-b378efb61762-host-slash\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-systemd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e71134f-cb26-40d1-90e4-50a4a79e40e1-host\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378095 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-conf\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1e71134f-cb26-40d1-90e4-50a4a79e40e1-serviceca\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378105 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e71134f-cb26-40d1-90e4-50a4a79e40e1-host\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-tuned\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feeb9bdc-2726-467c-a7e8-b378efb61762-host-slash\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-env-overrides\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-node-log\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-sysctl-conf\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-log-socket\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-netd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-device-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378344 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/feeb9bdc-2726-467c-a7e8-b378efb61762-iptables-alerter-script\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-dbus\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591f7c33-2794-4cba-bc61-4b4ec073028a-tmp-dir\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-log-socket\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.379238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378411 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-netd\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-device-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378410 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e67fa4a7-0013-40fb-b97b-9ab58fa62120-dbus\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378573 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-script-lib\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pzx\" (UniqueName: \"kubernetes.io/projected/591f7c33-2794-4cba-bc61-4b4ec073028a-kube-api-access-x9pzx\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-os-release\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-binary-copy\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-konnectivity-ca\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-os-release\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591f7c33-2794-4cba-bc61-4b4ec073028a-tmp-dir\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.378999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-kubernetes\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-run-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379042 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379070 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-kubernetes\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379107 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.380121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-agent-certs\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-kubelet\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm9c\" (UniqueName: \"kubernetes.io/projected/87d262f4-0010-4194-aa8d-96d95f1098bb-kube-api-access-hhm9c\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphsr\" (UniqueName: \"kubernetes.io/projected/1e71134f-cb26-40d1-90e4-50a4a79e40e1-kube-api-access-xphsr\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-run\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-lib-modules\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-etc-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-bin\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-binary-copy\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379461 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-kubelet\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-lib-modules\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-host-cni-bin\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-etc-openvswitch\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovn-node-metrics-cert\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.381134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-run\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-sys-fs\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-system-cni-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379961 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08a52778-2bd8-4b7b-92c6-003f40ae714b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.379998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-cnibin\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-system-cni-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-systemd-units\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-sys-fs\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-socket-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380159 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87d262f4-0010-4194-aa8d-96d95f1098bb-socket-dir\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34ce0d84-c5c9-49b2-8a0f-9384349b4986-systemd-units\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-cnibin\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380252 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/591f7c33-2794-4cba-bc61-4b4ec073028a-hosts-file\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-tmp\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.382028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380325 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/591f7c33-2794-4cba-bc61-4b4ec073028a-hosts-file\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82g2\" (UniqueName: \"kubernetes.io/projected/08a52778-2bd8-4b7b-92c6-003f40ae714b-kube-api-access-g82g2\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-host\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-config\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380522 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ee4fffa-2034-4aad-9eca-4eeb515321c7-host\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08a52778-2bd8-4b7b-92c6-003f40ae714b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-etc-tuned\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.380946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34ce0d84-c5c9-49b2-8a0f-9384349b4986-ovnkube-config\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.382230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e5663e5-0741-4934-8dd8-5fae3ad0da7a-agent-certs\") pod \"konnectivity-agent-6p989\" (UID: \"1e5663e5-0741-4934-8dd8-5fae3ad0da7a\") " pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.382768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.382482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ee4fffa-2034-4aad-9eca-4eeb515321c7-tmp\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.386630 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.386611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbd52\" (UniqueName: \"kubernetes.io/projected/34ce0d84-c5c9-49b2-8a0f-9384349b4986-kube-api-access-tbd52\") pod \"ovnkube-node-n5gv9\" (UID: \"34ce0d84-c5c9-49b2-8a0f-9384349b4986\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.391400 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.391376 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rkx\" (UniqueName: \"kubernetes.io/projected/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-kube-api-access-g4rkx\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.391784 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.391757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jjh\" (UniqueName: \"kubernetes.io/projected/feeb9bdc-2726-467c-a7e8-b378efb61762-kube-api-access-p2jjh\") pod \"iptables-alerter-6lsjj\" (UID: \"feeb9bdc-2726-467c-a7e8-b378efb61762\") " pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.391883 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.391837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphsr\" (UniqueName: \"kubernetes.io/projected/1e71134f-cb26-40d1-90e4-50a4a79e40e1-kube-api-access-xphsr\") pod \"node-ca-4qh9r\" (UID: \"1e71134f-cb26-40d1-90e4-50a4a79e40e1\") " pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.392114 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.392095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm9c\" (UniqueName: \"kubernetes.io/projected/87d262f4-0010-4194-aa8d-96d95f1098bb-kube-api-access-hhm9c\") pod \"aws-ebs-csi-driver-node-7wfcz\" (UID: \"87d262f4-0010-4194-aa8d-96d95f1098bb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.394901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.394871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzntr\" (UniqueName: \"kubernetes.io/projected/3ee4fffa-2034-4aad-9eca-4eeb515321c7-kube-api-access-rzntr\") pod \"tuned-vxglw\" (UID: \"3ee4fffa-2034-4aad-9eca-4eeb515321c7\") " pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.396603 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.395844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82g2\" (UniqueName: \"kubernetes.io/projected/08a52778-2bd8-4b7b-92c6-003f40ae714b-kube-api-access-g82g2\") pod \"multus-additional-cni-plugins-vwfwh\" (UID: \"08a52778-2bd8-4b7b-92c6-003f40ae714b\") " pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.397315 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.397295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pzx\" (UniqueName: \"kubernetes.io/projected/591f7c33-2794-4cba-bc61-4b4ec073028a-kube-api-access-x9pzx\") pod \"node-resolver-g9dld\" (UID: \"591f7c33-2794-4cba-bc61-4b4ec073028a\") " pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.414357 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.414323 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce85c6809f8a3cdc882d4b65d5c5ac9d.slice/crio-9d993dc79a9e3a5f4fa1e94cd55c0c6544628d9274deb599ca63077bc28ef503 WatchSource:0}: Error finding container 9d993dc79a9e3a5f4fa1e94cd55c0c6544628d9274deb599ca63077bc28ef503: Status 404 returned error can't find the container with id 9d993dc79a9e3a5f4fa1e94cd55c0c6544628d9274deb599ca63077bc28ef503 Jul 28 10:23:24.416106 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.415615 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5b667bb6865c4baa8a162abed9eb68.slice/crio-21d1663accb3a5ec68bef7cce17c58ef2c18efa24281d42841356c1adfdaccbc WatchSource:0}: Error finding container 21d1663accb3a5ec68bef7cce17c58ef2c18efa24281d42841356c1adfdaccbc: Status 404 returned error can't find the container with id 21d1663accb3a5ec68bef7cce17c58ef2c18efa24281d42841356c1adfdaccbc Jul 28 10:23:24.422056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.422040 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:23:24.487837 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.487815 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j99fk" Jul 28 10:23:24.497809 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.497784 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9a54f1_ccda_48b8_8a98_846d4781f112.slice/crio-b1a6c27e687f0d4b809d657282a02bead2e1df2eac81cb9abd0ef9f08cc8a20f WatchSource:0}: Error finding container b1a6c27e687f0d4b809d657282a02bead2e1df2eac81cb9abd0ef9f08cc8a20f: Status 404 returned error can't find the container with id b1a6c27e687f0d4b809d657282a02bead2e1df2eac81cb9abd0ef9f08cc8a20f Jul 28 10:23:24.515392 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.515369 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:24.521747 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.521720 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" Jul 28 10:23:24.523727 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.523706 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ce0d84_c5c9_49b2_8a0f_9384349b4986.slice/crio-0b3291ea6b3e61cc650dd6304e507aade771c0f6d73f01e8bdb75cc6c4812466 WatchSource:0}: Error finding container 0b3291ea6b3e61cc650dd6304e507aade771c0f6d73f01e8bdb75cc6c4812466: Status 404 returned error can't find the container with id 0b3291ea6b3e61cc650dd6304e507aade771c0f6d73f01e8bdb75cc6c4812466 Jul 28 10:23:24.524367 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.524350 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qh9r" Jul 28 10:23:24.531634 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.531610 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d262f4_0010_4194_aa8d_96d95f1098bb.slice/crio-9e434b2e14bc4cf92b795baa1f87ea7a9401f8e0a04e226857825e85415b9f24 WatchSource:0}: Error finding container 9e434b2e14bc4cf92b795baa1f87ea7a9401f8e0a04e226857825e85415b9f24: Status 404 returned error can't find the container with id 9e434b2e14bc4cf92b795baa1f87ea7a9401f8e0a04e226857825e85415b9f24 Jul 28 10:23:24.534246 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.534226 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e71134f_cb26_40d1_90e4_50a4a79e40e1.slice/crio-836b9a157b0e17c5c45474a7be85816de8e28b8e77256b9db0efca2f85daab77 WatchSource:0}: Error finding container 836b9a157b0e17c5c45474a7be85816de8e28b8e77256b9db0efca2f85daab77: Status 404 returned error can't find the container with id 836b9a157b0e17c5c45474a7be85816de8e28b8e77256b9db0efca2f85daab77 Jul 28 10:23:24.546079 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.546063 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6lsjj" Jul 28 10:23:24.556475 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.556452 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeeb9bdc_2726_467c_a7e8_b378efb61762.slice/crio-2f6a95d2970079276962fd5da461a4a0a7c5f9a072b146841825b81ea1ecf36b WatchSource:0}: Error finding container 2f6a95d2970079276962fd5da461a4a0a7c5f9a072b146841825b81ea1ecf36b: Status 404 returned error can't find the container with id 2f6a95d2970079276962fd5da461a4a0a7c5f9a072b146841825b81ea1ecf36b Jul 28 10:23:24.564253 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.564234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:24.574435 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.574412 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5663e5_0741_4934_8dd8_5fae3ad0da7a.slice/crio-b31b7dc4fdc440cabf50d2ea3dbc68a74f993e7c816032c631137fc23491c7fc WatchSource:0}: Error finding container b31b7dc4fdc440cabf50d2ea3dbc68a74f993e7c816032c631137fc23491c7fc: Status 404 returned error can't find the container with id b31b7dc4fdc440cabf50d2ea3dbc68a74f993e7c816032c631137fc23491c7fc Jul 28 10:23:24.581440 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.581415 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxglw" Jul 28 10:23:24.589145 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.589125 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee4fffa_2034_4aad_9eca_4eeb515321c7.slice/crio-6b3c281c67969239510f21e648ef66d022c5b5eb2ab7560ff7666edfa1012e6b WatchSource:0}: Error finding container 6b3c281c67969239510f21e648ef66d022c5b5eb2ab7560ff7666edfa1012e6b: Status 404 returned error can't find the container with id 6b3c281c67969239510f21e648ef66d022c5b5eb2ab7560ff7666edfa1012e6b Jul 28 10:23:24.589731 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.589702 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9dld" Jul 28 10:23:24.595765 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.595735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" Jul 28 10:23:24.598595 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.598575 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591f7c33_2794_4cba_bc61_4b4ec073028a.slice/crio-a15bd96ee38a6c6a8ad596eb1bd12e560f46c99d8f4e0dd96eb5d5614661e8ac WatchSource:0}: Error finding container a15bd96ee38a6c6a8ad596eb1bd12e560f46c99d8f4e0dd96eb5d5614661e8ac: Status 404 returned error can't find the container with id a15bd96ee38a6c6a8ad596eb1bd12e560f46c99d8f4e0dd96eb5d5614661e8ac Jul 28 10:23:24.604355 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:23:24.604336 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a52778_2bd8_4b7b_92c6_003f40ae714b.slice/crio-8757a95e51c9cac975d301e6505172d2028e983b1badffd0c25e11a49a227e45 WatchSource:0}: Error finding container 8757a95e51c9cac975d301e6505172d2028e983b1badffd0c25e11a49a227e45: Status 404 returned error can't find the container with id 8757a95e51c9cac975d301e6505172d2028e983b1badffd0c25e11a49a227e45 Jul 28 10:23:24.606107 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.606084 2567 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-vwfwh_openshift-multus(08a52778-2bd8-4b7b-92c6-003f40ae714b): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 28 10:23:24.606175 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.606125 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" podUID="08a52778-2bd8-4b7b-92c6-003f40ae714b" Jul 28 10:23:24.784022 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.783923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:24.784195 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.784086 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:24.784195 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.784113 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:24.784195 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.784128 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:24.784338 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.784196 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:25.784175365 +0000 UTC m=+2.916834624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:24.884982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.884939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:24.885157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:24.884994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:24.885213 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.885161 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:24.885262 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.885220 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:25.885199296 +0000 UTC m=+3.017858565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:24.885632 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.885615 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:24.885707 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:24.885679 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:25.885653105 +0000 UTC m=+3.018312368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:25.217328 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.217202 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:18:24 +0000 UTC" deadline="2028-02-17 06:59:48.706028 +0000 UTC" Jul 28 10:23:25.217328 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.217266 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13652h36m23.48876587s" Jul 28 10:23:25.381041 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.381009 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:25.381238 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.381152 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:25.424545 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.424476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9dld" event={"ID":"591f7c33-2794-4cba-bc61-4b4ec073028a","Type":"ContainerStarted","Data":"a15bd96ee38a6c6a8ad596eb1bd12e560f46c99d8f4e0dd96eb5d5614661e8ac"} Jul 28 10:23:25.432848 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.428240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxglw" event={"ID":"3ee4fffa-2034-4aad-9eca-4eeb515321c7","Type":"ContainerStarted","Data":"6b3c281c67969239510f21e648ef66d022c5b5eb2ab7560ff7666edfa1012e6b"} Jul 28 10:23:25.443043 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.443010 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6p989" event={"ID":"1e5663e5-0741-4934-8dd8-5fae3ad0da7a","Type":"ContainerStarted","Data":"b31b7dc4fdc440cabf50d2ea3dbc68a74f993e7c816032c631137fc23491c7fc"} Jul 28 10:23:25.461148 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.461117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6lsjj" event={"ID":"feeb9bdc-2726-467c-a7e8-b378efb61762","Type":"ContainerStarted","Data":"2f6a95d2970079276962fd5da461a4a0a7c5f9a072b146841825b81ea1ecf36b"} Jul 28 10:23:25.474241 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.474122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qh9r" event={"ID":"1e71134f-cb26-40d1-90e4-50a4a79e40e1","Type":"ContainerStarted","Data":"836b9a157b0e17c5c45474a7be85816de8e28b8e77256b9db0efca2f85daab77"} Jul 28 10:23:25.505499 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.505462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" event={"ID":"87d262f4-0010-4194-aa8d-96d95f1098bb","Type":"ContainerStarted","Data":"9e434b2e14bc4cf92b795baa1f87ea7a9401f8e0a04e226857825e85415b9f24"} Jul 28 10:23:25.525090 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.525057 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j99fk" event={"ID":"2d9a54f1-ccda-48b8-8a98-846d4781f112","Type":"ContainerStarted","Data":"b1a6c27e687f0d4b809d657282a02bead2e1df2eac81cb9abd0ef9f08cc8a20f"} Jul 28 10:23:25.572312 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.572274 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" event={"ID":"ce85c6809f8a3cdc882d4b65d5c5ac9d","Type":"ContainerStarted","Data":"9d993dc79a9e3a5f4fa1e94cd55c0c6544628d9274deb599ca63077bc28ef503"} Jul 28 10:23:25.594519 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.594481 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerStarted","Data":"8757a95e51c9cac975d301e6505172d2028e983b1badffd0c25e11a49a227e45"} Jul 28 10:23:25.601639 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.601607 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:25.604251 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.604216 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" podUID="08a52778-2bd8-4b7b-92c6-003f40ae714b" Jul 28 10:23:25.626636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.626591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"0b3291ea6b3e61cc650dd6304e507aade771c0f6d73f01e8bdb75cc6c4812466"} Jul 28 10:23:25.648881 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.648696 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"6a5b667bb6865c4baa8a162abed9eb68","Type":"ContainerStarted","Data":"21d1663accb3a5ec68bef7cce17c58ef2c18efa24281d42841356c1adfdaccbc"} Jul 28 10:23:25.790470 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.790377 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:25.793081 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.793055 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:25.793227 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.793210 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:25.793295 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.793229 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:25.793295 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.793256 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:25.793395 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.793311 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:27.793291906 +0000 UTC m=+4.925951166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:25.893769 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.893729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:25.893956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:25.893793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:25.893956 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.893906 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:25.894056 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.893968 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:27.893950537 +0000 UTC m=+5.026609798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:25.894491 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.894468 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:25.894613 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:25.894526 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:27.894510538 +0000 UTC m=+5.027169800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:26.217526 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:26.217456 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:18:24 +0000 UTC" deadline="2028-04-08 22:22:49.76934301 +0000 UTC" Jul 28 10:23:26.217526 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:26.217524 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14891h59m23.55182358s" Jul 28 10:23:26.381376 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:26.381345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:26.381566 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:26.381481 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:26.381927 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:26.381905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:26.382055 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:26.382031 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:26.834015 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:26.833725 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:23:27.383898 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:27.382000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:27.383898 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.382173 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:27.811208 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:27.811165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:27.811408 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.811348 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:27.811408 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.811374 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:27.811408 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.811389 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:27.811591 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.811456 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:31.811434398 +0000 UTC m=+8.944093691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:27.912058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:27.912009 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:27.912249 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:27.912082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:27.912249 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.912241 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:27.912381 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.912310 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:31.912289214 +0000 UTC m=+9.044948481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:27.912886 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.912865 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:27.912972 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:27.912933 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:31.912913569 +0000 UTC m=+9.045572834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:28.381934 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:28.381242 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:28.381934 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:28.381378 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:28.381934 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:28.381452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:28.381934 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:28.381536 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:29.380866 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:29.380832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:29.381323 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:29.380974 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:30.380870 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:30.380809 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:30.381250 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:30.380965 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:30.381250 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:30.381027 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:30.381250 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:30.381105 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:31.381279 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:31.380765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:31.381279 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.380911 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:31.847582 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:31.847541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:31.847781 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.847762 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:31.847876 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.847789 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:31.847876 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.847802 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:31.847876 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.847858 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:39.847840682 +0000 UTC m=+16.980499942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:31.948242 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:31.948209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:31.948432 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:31.948258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:31.948432 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.948370 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:31.948432 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.948422 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:39.948408971 +0000 UTC m=+17.081068228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:31.948761 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.948727 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:31.948854 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:31.948786 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:39.948770703 +0000 UTC m=+17.081429960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:32.381151 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:32.381109 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:32.381339 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:32.381224 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:32.381715 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:32.381631 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:32.381818 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:32.381756 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:33.383583 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:33.383501 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:33.384034 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:33.383612 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:34.380567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:34.380534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:34.380755 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:34.380679 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:34.380755 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:34.380708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:34.380878 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:34.380835 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:35.380650 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:35.380614 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:35.381116 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:35.380713 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:36.381497 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:36.381457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:36.381926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:36.381457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:36.381926 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:36.381589 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:36.381926 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:36.381707 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:37.380675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:37.380636 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:37.380872 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:37.380769 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:38.381223 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:38.381189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:38.381625 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:38.381227 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:38.381625 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:38.381298 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:38.381625 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:38.381441 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:39.380846 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:39.380808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:39.381068 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:39.380932 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:39.911221 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:39.911190 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:39.911647 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:39.911315 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:39.911647 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:39.911329 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:39.911647 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:39.911337 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:39.911647 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:39.911378 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:55.911366364 +0000 UTC m=+33.044025621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:40.011907 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:40.011873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:40.012067 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:40.011918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:40.012067 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.012044 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:40.012067 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.012048 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:40.012204 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.012105 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:56.012088021 +0000 UTC m=+33.144747278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:40.012204 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.012123 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:56.012113778 +0000 UTC m=+33.144773036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:40.380867 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:40.380839 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:40.381022 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:40.380839 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:40.381022 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.380973 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:40.381120 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:40.381095 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:41.381454 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:41.381425 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:41.381873 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:41.381700 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:42.381330 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:42.381299 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:42.381509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:42.381300 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:42.381509 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:42.381412 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:42.381509 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:42.381501 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:43.381270 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:43.381222 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:43.381456 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:43.381323 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:44.381353 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.381153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:44.381850 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.381201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:44.381850 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:44.381465 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:44.381850 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:44.381543 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:44.691132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.691039 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxglw" event={"ID":"3ee4fffa-2034-4aad-9eca-4eeb515321c7","Type":"ContainerStarted","Data":"206e34e1dc1140fff891ecb9e692a2acb1dd1b9c012afbf71278e0df97732c41"} Jul 28 10:23:44.696275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.694581 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j99fk" event={"ID":"2d9a54f1-ccda-48b8-8a98-846d4781f112","Type":"ContainerStarted","Data":"66d56c7e0839f776a54763c49d55fca1e81d1f2e7ec75c152f9ec1c703d99e4e"} Jul 28 10:23:44.700712 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.700597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" event={"ID":"ce85c6809f8a3cdc882d4b65d5c5ac9d","Type":"ContainerStarted","Data":"71fef9a9020413716df7848e93fdf27eb779f40d104352d4e50e1d33dd668ef5"} Jul 28 10:23:44.717697 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.717352 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vxglw" podStartSLOduration=2.294886133 podStartE2EDuration="21.717336076s" totalImagesPullingTime="19.422449943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:44.717076225 +0000 UTC m=+21.849735504" watchObservedRunningTime="2026-07-28 10:23:44.717336076 +0000 UTC m=+21.849995359" Jul 28 10:23:44.738349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.736484 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-55.ec2.internal" podStartSLOduration=20.736465476 podStartE2EDuration="20.736465476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:44.735240137 +0000 UTC m=+21.867899417" watchObservedRunningTime="2026-07-28 10:23:44.736465476 +0000 UTC m=+21.869124756" Jul 28 10:23:44.753338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:44.753286 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j99fk" podStartSLOduration=2.197475979 podStartE2EDuration="21.753264416s" totalImagesPullingTime="19.555788437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:44.753242322 +0000 UTC m=+21.885901601" watchObservedRunningTime="2026-07-28 10:23:44.753264416 +0000 UTC m=+21.885923696" Jul 28 10:23:45.381046 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.380837 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:45.381285 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:45.381134 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:45.704501 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.704473 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"8dd087e837379234f566b7a974126df98edce56a86d621fa8ad453d269328b3a"} Jul 28 10:23:45.705104 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.704547 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"8c87ecbf772f4cc83c2e315ed575b2e32c1cfa732eba4b9a7c1904a3667da910"} Jul 28 10:23:45.705104 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.704588 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"56762584fcd68facbdec159f1a1c5afc41d7ddc11fcae0f0d8f0ab3fad48be5e"} Jul 28 10:23:45.705104 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.704624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"e33e8b6907751f4e6988aadbc1dece6dcc630f7230933d51f91e2b00235ed4e0"} Jul 28 10:23:45.705764 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.705724 2567 generic.go:356] "Generic (PLEG): container finished" podID="6a5b667bb6865c4baa8a162abed9eb68" containerID="222620724f426fc2cbda64c22fb21680eb8e6ca957bde38ff594c796d13032e9" exitCode=0 Jul 28 10:23:45.705849 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.705768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"6a5b667bb6865c4baa8a162abed9eb68","Type":"ContainerDied","Data":"222620724f426fc2cbda64c22fb21680eb8e6ca957bde38ff594c796d13032e9"} Jul 28 10:23:45.707236 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.707217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9dld" event={"ID":"591f7c33-2794-4cba-bc61-4b4ec073028a","Type":"ContainerStarted","Data":"75076ee0e57c4b355b97250eb502b231f1d9499555b5915d3f0323279f67dfd6"} Jul 28 10:23:45.708538 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.708511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6p989" event={"ID":"1e5663e5-0741-4934-8dd8-5fae3ad0da7a","Type":"ContainerStarted","Data":"8334aa6abe35216b25f34cc8fae14c68616ffa7fecf7dfcebd17e6a3954a38df"} Jul 28 10:23:45.709934 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.709918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6lsjj" event={"ID":"feeb9bdc-2726-467c-a7e8-b378efb61762","Type":"ContainerStarted","Data":"8c18e65334beb5a019c7a737fdf6fb2c1709b991c343c074ea31571573d65bc2"} Jul 28 10:23:45.711409 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.711385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qh9r" event={"ID":"1e71134f-cb26-40d1-90e4-50a4a79e40e1","Type":"ContainerStarted","Data":"1f4665f31d46c11170ad1b4e9e3183d0895682abb92a1491a63105229e607f26"} Jul 28 10:23:45.712701 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.712683 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" event={"ID":"87d262f4-0010-4194-aa8d-96d95f1098bb","Type":"ContainerStarted","Data":"ceb89a64cc8092de96f4a7d3bb24ffb23796a2e0782b581c18ecba72a2ceaf0e"} Jul 28 10:23:45.737634 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.737478 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9dld_591f7c33-2794-4cba-bc61-4b4ec073028a/dns-node-resolver/0.log" Jul 28 10:23:45.752589 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.752542 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6p989" podStartSLOduration=11.233156975 podStartE2EDuration="22.752524634s" totalImagesPullingTime="11.519367659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:45.752180012 +0000 UTC m=+22.884839291" watchObservedRunningTime="2026-07-28 10:23:45.752524634 +0000 UTC m=+22.885183915" Jul 28 10:23:45.753223 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.753185 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4qh9r" podStartSLOduration=3.26125289 podStartE2EDuration="22.753175367s" totalImagesPullingTime="19.491922477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:45.73501726 +0000 UTC m=+22.867676538" watchObservedRunningTime="2026-07-28 10:23:45.753175367 +0000 UTC m=+22.885834649" Jul 28 10:23:45.768110 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.768068 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g9dld" podStartSLOduration=3.35599168 podStartE2EDuration="22.768052579s" totalImagesPullingTime="19.412060899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:45.767920945 +0000 UTC m=+22.900580223" watchObservedRunningTime="2026-07-28 10:23:45.768052579 +0000 UTC m=+22.900711920" Jul 28 10:23:45.777172 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.777152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:45.782359 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.782330 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6lsjj" podStartSLOduration=7.456221404 podStartE2EDuration="22.782319957s" totalImagesPullingTime="15.326098553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:45.781719019 +0000 UTC m=+22.914378300" watchObservedRunningTime="2026-07-28 10:23:45.782319957 +0000 UTC m=+22.914979235" Jul 28 10:23:45.919252 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:45.919221 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:46.013118 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.013093 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:46.377917 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.377887 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:46.380585 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.380568 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:46.380683 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:46.380667 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:46.380774 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.380756 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:46.380881 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:46.380864 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:46.408221 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.408198 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:46.408633 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.408618 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:46.511909 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.511886 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qh9r_1e71134f-cb26-40d1-90e4-50a4a79e40e1/node-ca/0.log" Jul 28 10:23:46.654697 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.654674 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 10:23:46.716799 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.716768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" event={"ID":"87d262f4-0010-4194-aa8d-96d95f1098bb","Type":"ContainerStarted","Data":"012793b3f51322afdaa9087277f8d93e73bc9ec2cfc71b0f71656ff62e8ea627"} Jul 28 10:23:46.718402 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.718380 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="56842874f944a82250f7dc282114deabaace0371e4bcdd2728832125b9d7dc23" exitCode=0 Jul 28 10:23:46.718507 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.718467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"56842874f944a82250f7dc282114deabaace0371e4bcdd2728832125b9d7dc23"} Jul 28 10:23:46.721614 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.721595 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"6ff5ae004b10218cf6afd3d6c6c5b1729f9788ef3058f807971c0448c3036c4c"} Jul 28 10:23:46.721686 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.721652 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"81b50c4dec9be76578b5e1fdaa980e3a9ff7e63adb81d6e49aab15e51cf90a5d"} Jul 28 10:23:46.728886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.728862 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" event={"ID":"6a5b667bb6865c4baa8a162abed9eb68","Type":"ContainerStarted","Data":"ed248a69e8351f30018caff03cbf9ac68923b033d49e4ac4c75e6234f67c9ad9"} Jul 28 10:23:46.729517 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.729492 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:46.729616 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.729552 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6p989" Jul 28 10:23:46.773195 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.773156 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-55.ec2.internal" podStartSLOduration=22.77314469 podStartE2EDuration="22.77314469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:46.773038692 +0000 UTC m=+23.905697971" watchObservedRunningTime="2026-07-28 10:23:46.77314469 +0000 UTC m=+23.905803969" Jul 28 10:23:46.807294 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:46.807266 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:47.244999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.244891 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:23:46.654693774Z","UUID":"364b9a92-4351-449c-92aa-38aeff970e80","Handler":null,"Name":"","Endpoint":""} Jul 28 10:23:47.246444 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.246423 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 10:23:47.246545 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.246460 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 10:23:47.336438 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.336417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:47.380512 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.380490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:47.380648 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:47.380607 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:47.421211 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.421187 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:47.859752 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.859711 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:47.960915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:47.960883 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:48.272870 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.272839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:48.380781 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.380748 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:48.380943 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:48.380847 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:48.380943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.380914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:48.381032 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:48.380997 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:48.736315 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.736279 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" event={"ID":"87d262f4-0010-4194-aa8d-96d95f1098bb","Type":"ContainerStarted","Data":"3160cdf71c008ac801796f433119ebbb7e3c18d46722f3005c24a850c440e81e"} Jul 28 10:23:48.740099 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.740073 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"01f313ac3b8909d62696f422f60929507bd2e8110c7abf3d720bd3e1feec8672"} Jul 28 10:23:48.761432 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:48.761391 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wfcz" podStartSLOduration=3.425479338 podStartE2EDuration="25.761374048s" totalImagesPullingTime="22.33589471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:48.761084804 +0000 UTC m=+25.893744087" watchObservedRunningTime="2026-07-28 10:23:48.761374048 +0000 UTC m=+25.894033328" Jul 28 10:23:49.381550 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:49.381516 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:49.381989 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:49.381640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:50.372181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.372111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:50.381068 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.381038 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:50.381201 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.381044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:50.381201 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:50.381161 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:50.381297 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:50.381214 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:50.471752 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.471715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:50.573565 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.573530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:50.673520 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.673458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:50.773891 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.773864 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:50.879196 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:50.879166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:51.080315 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.080141 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:51.381024 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.381000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:51.381121 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:51.381108 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:51.748124 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.748100 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="8d966aa8e4063da07c2b09751a199918a4c1b0161686865485dd67ea2f7311e9" exitCode=0 Jul 28 10:23:51.748957 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.748172 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"8d966aa8e4063da07c2b09751a199918a4c1b0161686865485dd67ea2f7311e9"} Jul 28 10:23:51.752108 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.752089 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" event={"ID":"34ce0d84-c5c9-49b2-8a0f-9384349b4986","Type":"ContainerStarted","Data":"cd9495e615357a71b4bf980512cd504f05fc36ebf37f7e279eaf5b144792ed96"} Jul 28 10:23:51.752362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.752342 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:51.752457 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.752404 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:51.767828 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.767808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:51.803983 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.803950 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" podStartSLOduration=8.730333434 podStartE2EDuration="28.803938216s" totalImagesPullingTime="20.073604782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:51.802581674 +0000 UTC m=+28.935240954" watchObservedRunningTime="2026-07-28 10:23:51.803938216 +0000 UTC m=+28.936597527" Jul 28 10:23:51.871981 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:51.871954 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:52.069835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.069777 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:52.177484 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.177460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:52.380943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.380871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:52.380943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.380887 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:52.381138 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:52.380988 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:52.381138 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:52.381121 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:52.477390 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.477364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:52.754572 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.754546 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:52.769294 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:52.769275 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:23:53.079904 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.079848 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:53.382250 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.382193 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:53.382362 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:53.382300 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:53.450842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.450821 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:53.554758 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.554721 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:53.758081 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.758056 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="72d55861117a994717dcc8f75ddfbde151f201b189e7f5aa757ca2be836035b3" exitCode=0 Jul 28 10:23:53.758426 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.758124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"72d55861117a994717dcc8f75ddfbde151f201b189e7f5aa757ca2be836035b3"} Jul 28 10:23:53.866479 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:53.866451 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:54.263473 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.263442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:54.363351 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.363326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:54.381463 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.381444 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:54.381548 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.381470 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:54.381606 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:54.381547 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:54.381679 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:54.381658 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:54.470054 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.470024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:54.761916 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.761890 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="e7a6ba43d4ce7d80e75e755d66c2b1a5e50192f7a83cc1c31fcfefdac4396d78" exitCode=0 Jul 28 10:23:54.762194 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.761963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"e7a6ba43d4ce7d80e75e755d66c2b1a5e50192f7a83cc1c31fcfefdac4396d78"} Jul 28 10:23:54.861320 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.861293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:54.930436 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:54.930377 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:55.380978 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:55.380948 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:55.381162 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:55.381071 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:55.500956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:55.500922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:55.929126 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:55.929100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:55.929507 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:55.929220 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:23:55.929507 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:55.929234 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:23:55.929507 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:55.929242 2567 projected.go:196] Error preparing data for projected volume kube-api-access-5mvz8 for pod openshift-network-diagnostics/network-check-target-xnrgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:55.929507 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:55.929283 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8 podName:7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:27.929271322 +0000 UTC m=+65.061930578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mvz8" (UniqueName: "kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8") pod "network-check-target-xnrgz" (UID: "7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:23:56.030204 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.030175 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:56.030348 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.030212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:56.030348 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.030312 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:56.030348 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.030314 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:56.030508 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.030363 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs podName:8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c nodeName:}" failed. No retries permitted until 2026-07-28 10:24:28.030350323 +0000 UTC m=+65.163009581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs") pod "network-metrics-daemon-4st7m" (UID: "8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:23:56.030508 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.030398 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret podName:e67fa4a7-0013-40fb-b97b-9ab58fa62120 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:28.030379687 +0000 UTC m=+65.163038963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret") pod "global-pull-secret-syncer-swstv" (UID: "e67fa4a7-0013-40fb-b97b-9ab58fa62120") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:23:56.061219 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.061175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:56.262409 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.262364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:56.381209 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.381189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:56.381341 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.381286 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:56.381388 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.381368 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:56.381462 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:56.381444 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:56.464032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.464004 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:56.856553 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.856512 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:56.958068 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:56.958042 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:57.381172 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:57.381146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:57.381428 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:57.381264 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:57.661491 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:57.661414 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:58.079128 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.079099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:58.266311 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.266119 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:58.380467 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.380401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:23:58.380622 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.380401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:23:58.380622 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:58.380506 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:23:58.380622 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:58.380598 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:23:58.464140 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.464108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:58.868252 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:58.868215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:59.381503 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.381481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:23:59.381875 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:23:59.381614 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:23:59.506869 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.506845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:59.668606 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.668517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:23:59.775331 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.775306 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="d336e8d4d7fe80ea1a6888a7539c5f3ab5ab693f2caafcd4a58e28cb9afdd6b7" exitCode=0 Jul 28 10:23:59.775434 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.775353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"d336e8d4d7fe80ea1a6888a7539c5f3ab5ab693f2caafcd4a58e28cb9afdd6b7"} Jul 28 10:23:59.868395 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:23:59.868373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:00.270287 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.270255 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:00.381471 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.381439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:00.381471 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.381465 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:00.381868 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:00.381553 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:00.381868 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:00.381666 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:00.460617 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.460590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:00.780475 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.780449 2567 generic.go:356] "Generic (PLEG): container finished" podID="08a52778-2bd8-4b7b-92c6-003f40ae714b" containerID="5178b2e69eeead04259b6e010efa4898db7af4785e7b896590ad7aa1a86358c0" exitCode=0 Jul 28 10:24:00.780629 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.780519 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerDied","Data":"5178b2e69eeead04259b6e010efa4898db7af4785e7b896590ad7aa1a86358c0"} Jul 28 10:24:00.860838 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:00.860816 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:01.068305 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:01.068232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:01.381133 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:01.381070 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:01.381254 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:01.381185 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:01.785557 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:01.785529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" event={"ID":"08a52778-2bd8-4b7b-92c6-003f40ae714b","Type":"ContainerStarted","Data":"65f5e8f3d8dad759fa13ac9f33db18e3a8641775cc22558d2b1224cf57906903"} Jul 28 10:24:01.811204 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:01.811165 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vwfwh" podStartSLOduration=4.065674825 podStartE2EDuration="38.811151346s" totalImagesPullingTime="34.745476521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 10:24:01.810941683 +0000 UTC m=+38.943600987" watchObservedRunningTime="2026-07-28 10:24:01.811151346 +0000 UTC m=+38.943810624" Jul 28 10:24:02.380550 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:02.380523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:02.380724 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:02.380617 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:02.380724 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:02.380685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:02.380872 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:02.380812 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:03.382296 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:03.382261 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:03.382641 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:03.382370 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:03.998922 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:03.998888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:04.056366 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:04.056337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:04.381025 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:04.380956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:04.381164 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:04.380957 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:04.381164 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:04.381068 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:04.381164 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:04.381139 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:05.381499 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:05.381465 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:05.382022 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:05.381563 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:06.381185 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:06.381152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:06.381338 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:06.381249 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:06.381338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:06.381304 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:06.381414 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:06.381383 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:07.380976 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:07.380944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:07.381344 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:07.381066 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:08.307020 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.306986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:08.381366 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.381340 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:08.381787 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:08.381436 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:08.381787 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.381469 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:08.381787 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:08.381520 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:08.387501 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.387479 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:08.490387 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.490359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:08.586721 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.586647 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:08.687003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.686980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:08.744051 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:08.744024 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:09.381193 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:09.381166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:09.381342 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:09.381268 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:09.447925 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:09.447898 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:09.548772 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:09.548749 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:09.878338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:09.878307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:09.936749 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:09.936708 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:10.332087 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.332051 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xnrgz"] Jul 28 10:24:10.332240 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.332206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:10.332334 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:10.332309 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:10.336450 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.336426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4st7m"] Jul 28 10:24:10.336574 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.336559 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:10.336700 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:10.336676 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:10.337056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.337032 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swstv"] Jul 28 10:24:10.337154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:10.337126 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:10.337351 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:10.337207 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:11.283663 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:11.283633 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:11.455901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:11.455873 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:11.628498 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:11.628461 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:12.085486 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:12.085301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:12.185661 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:12.185610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:12.380611 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:12.380546 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:12.380611 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:12.380575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:12.381057 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:12.380554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:12.381057 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:12.380662 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swstv" podUID="e67fa4a7-0013-40fb-b97b-9ab58fa62120" Jul 28 10:24:12.381057 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:12.380725 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xnrgz" podUID="7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076" Jul 28 10:24:12.381057 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:12.380849 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4st7m" podUID="8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c" Jul 28 10:24:14.202419 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.202352 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-55.ec2.internal" event="NodeReady" Jul 28 10:24:14.202788 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.202449 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 10:24:14.236343 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.236310 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5879f5564b-9vffh"] Jul 28 10:24:14.253431 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.253410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5879f5564b-9vffh"] Jul 28 10:24:14.253557 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.253500 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.253837 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.253720 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nv8zj"] Jul 28 10:24:14.256204 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.256188 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 10:24:14.256317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.256188 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 10:24:14.256317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.256189 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f7477\"" Jul 28 10:24:14.256317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.256190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 10:24:14.261993 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.261865 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 10:24:14.275463 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.275443 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xp886"] Jul 28 10:24:14.275593 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.275577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.278156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.278137 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 10:24:14.278156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.278152 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f5cv\"" Jul 28 10:24:14.278312 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.278253 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 10:24:14.296442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.296423 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nv8zj"] Jul 28 10:24:14.296526 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.296461 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xp886"] Jul 28 10:24:14.296564 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.296531 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.299008 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.298989 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 10:24:14.299199 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.299182 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 10:24:14.299267 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.299233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9dkdj\"" Jul 28 10:24:14.299267 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.299254 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 10:24:14.355567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.355548 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2lsd5"] Jul 28 10:24:14.369047 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369025 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-bound-sa-token\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-tmp-dir\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.369117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369074 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-tls\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08fb446f-5afe-4551-9e6a-5a21956bc677-ca-trust-extracted\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdc44\" (UniqueName: \"kubernetes.io/projected/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-kube-api-access-mdc44\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.369246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369169 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-metrics-tls\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.369246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-trusted-ca\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-config-volume\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.369246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzrx\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-kube-api-access-6pzrx\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369360 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369268 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-certificates\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369360 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-installation-pull-secrets\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.369421 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.369366 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-image-registry-private-configuration\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.378355 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.378337 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2lsd5"] Jul 28 10:24:14.378435 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.378416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.380484 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.380387 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:14.380594 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.380520 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:14.380666 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.380648 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:14.381121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.381102 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 10:24:14.381214 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.381106 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kn9mk\"" Jul 28 10:24:14.381335 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.381323 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 10:24:14.381421 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.381406 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 10:24:14.381544 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.381527 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 10:24:14.382841 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.382826 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 10:24:14.382962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.382904 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mqqnj\"" Jul 28 10:24:14.382962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.382916 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 10:24:14.383157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.383139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8src\"" Jul 28 10:24:14.383274 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.383168 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 10:24:14.383522 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.383504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 10:24:14.390435 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.390416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:14.470156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-bound-sa-token\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-tmp-dir\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.470262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-tls\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08fb446f-5afe-4551-9e6a-5a21956bc677-ca-trust-extracted\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdc44\" (UniqueName: \"kubernetes.io/projected/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-kube-api-access-mdc44\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.470262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453b6262-344b-42ae-b448-629bce3a1a62-data-volume\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.470509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470278 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453b6262-344b-42ae-b448-629bce3a1a62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.470509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470422 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d5d4cd7-0d96-471b-894c-fe57fd572b01-cert\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.470509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqbk\" (UniqueName: \"kubernetes.io/projected/2d5d4cd7-0d96-471b-894c-fe57fd572b01-kube-api-access-jmqbk\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-tmp-dir\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-metrics-tls\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-trusted-ca\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470572 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453b6262-344b-42ae-b448-629bce3a1a62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-config-volume\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.470654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzrx\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-kube-api-access-6pzrx\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470946 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453b6262-344b-42ae-b448-629bce3a1a62-crio-socket\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.470946 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tr9c\" (UniqueName: \"kubernetes.io/projected/453b6262-344b-42ae-b448-629bce3a1a62-kube-api-access-5tr9c\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.470946 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-certificates\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470946 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470782 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-installation-pull-secrets\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.470946 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.470835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-image-registry-private-configuration\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.471151 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.471133 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-config-volume\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.474417 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.474399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-metrics-tls\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.474497 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.474441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-tls\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.478768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.478722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdc44\" (UniqueName: \"kubernetes.io/projected/0ff2fb62-a7e2-4295-aceb-6fdeef1993b9-kube-api-access-mdc44\") pod \"dns-default-nv8zj\" (UID: \"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9\") " pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.478894 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.478729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-bound-sa-token\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.478894 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.478880 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzrx\" (UniqueName: \"kubernetes.io/projected/08fb446f-5afe-4551-9e6a-5a21956bc677-kube-api-access-6pzrx\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.484159 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.484087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08fb446f-5afe-4551-9e6a-5a21956bc677-ca-trust-extracted\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.484588 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.484542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-registry-certificates\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.485226 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.485193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:14.485321 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.485252 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08fb446f-5afe-4551-9e6a-5a21956bc677-trusted-ca\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.486122 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.486099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-installation-pull-secrets\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.486432 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.486410 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08fb446f-5afe-4551-9e6a-5a21956bc677-image-registry-private-configuration\") pod \"image-registry-5879f5564b-9vffh\" (UID: \"08fb446f-5afe-4551-9e6a-5a21956bc677\") " pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.564379 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.564355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:14.571588 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453b6262-344b-42ae-b448-629bce3a1a62-data-volume\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453b6262-344b-42ae-b448-629bce3a1a62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d5d4cd7-0d96-471b-894c-fe57fd572b01-cert\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.571667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571643 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqbk\" (UniqueName: \"kubernetes.io/projected/2d5d4cd7-0d96-471b-894c-fe57fd572b01-kube-api-access-jmqbk\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.571865 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571685 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453b6262-344b-42ae-b448-629bce3a1a62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571865 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453b6262-344b-42ae-b448-629bce3a1a62-crio-socket\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571865 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tr9c\" (UniqueName: \"kubernetes.io/projected/453b6262-344b-42ae-b448-629bce3a1a62-kube-api-access-5tr9c\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/453b6262-344b-42ae-b448-629bce3a1a62-crio-socket\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.571999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.571945 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/453b6262-344b-42ae-b448-629bce3a1a62-data-volume\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.572202 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.572182 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/453b6262-344b-42ae-b448-629bce3a1a62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.573969 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.573945 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/453b6262-344b-42ae-b448-629bce3a1a62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.574033 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.574003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d5d4cd7-0d96-471b-894c-fe57fd572b01-cert\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.579661 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.579643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tr9c\" (UniqueName: \"kubernetes.io/projected/453b6262-344b-42ae-b448-629bce3a1a62-kube-api-access-5tr9c\") pod \"insights-runtime-extractor-2lsd5\" (UID: \"453b6262-344b-42ae-b448-629bce3a1a62\") " pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.579905 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.579874 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqbk\" (UniqueName: \"kubernetes.io/projected/2d5d4cd7-0d96-471b-894c-fe57fd572b01-kube-api-access-jmqbk\") pod \"ingress-canary-xp886\" (UID: \"2d5d4cd7-0d96-471b-894c-fe57fd572b01\") " pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.583480 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.583462 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:14.584332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.584308 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:14.605880 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.605851 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xp886" Jul 28 10:24:14.689959 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.689617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2lsd5" Jul 28 10:24:14.768992 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.768727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5879f5564b-9vffh"] Jul 28 10:24:14.772878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.772848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xp886"] Jul 28 10:24:14.772985 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.772911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nv8zj"] Jul 28 10:24:14.779326 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:14.779300 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff2fb62_a7e2_4295_aceb_6fdeef1993b9.slice/crio-0db86ee4fdb2d0b84ac0ede10ba83a205f01db80e8aec05727e5b575dbc6f311 WatchSource:0}: Error finding container 0db86ee4fdb2d0b84ac0ede10ba83a205f01db80e8aec05727e5b575dbc6f311: Status 404 returned error can't find the container with id 0db86ee4fdb2d0b84ac0ede10ba83a205f01db80e8aec05727e5b575dbc6f311 Jul 28 10:24:14.780003 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:14.779983 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5d4cd7_0d96_471b_894c_fe57fd572b01.slice/crio-9a9cc420db8d769ecf559b30d2b079bb3749adf1d2120149f62e4f8d5bfaf725 WatchSource:0}: Error finding container 9a9cc420db8d769ecf559b30d2b079bb3749adf1d2120149f62e4f8d5bfaf725: Status 404 returned error can't find the container with id 9a9cc420db8d769ecf559b30d2b079bb3749adf1d2120149f62e4f8d5bfaf725 Jul 28 10:24:14.814108 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.813905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xp886" event={"ID":"2d5d4cd7-0d96-471b-894c-fe57fd572b01","Type":"ContainerStarted","Data":"9a9cc420db8d769ecf559b30d2b079bb3749adf1d2120149f62e4f8d5bfaf725"} Jul 28 10:24:14.815024 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.815002 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nv8zj" event={"ID":"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9","Type":"ContainerStarted","Data":"0db86ee4fdb2d0b84ac0ede10ba83a205f01db80e8aec05727e5b575dbc6f311"} Jul 28 10:24:14.816317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.816299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" event={"ID":"08fb446f-5afe-4551-9e6a-5a21956bc677","Type":"ContainerStarted","Data":"7b4f3efb17a6f912b51248deb2bbf859c5c043aae1af1cd002e93d950c689ebb"} Jul 28 10:24:14.830581 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:14.830517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2lsd5"] Jul 28 10:24:14.833746 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:14.833712 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453b6262_344b_42ae_b448_629bce3a1a62.slice/crio-f63e35988cbf35094cc8d37542c9543cd16f3c62e7cd23df5775b410beaad850 WatchSource:0}: Error finding container f63e35988cbf35094cc8d37542c9543cd16f3c62e7cd23df5775b410beaad850: Status 404 returned error can't find the container with id f63e35988cbf35094cc8d37542c9543cd16f3c62e7cd23df5775b410beaad850 Jul 28 10:24:15.821819 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:15.821690 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2lsd5" event={"ID":"453b6262-344b-42ae-b448-629bce3a1a62","Type":"ContainerStarted","Data":"f63e35988cbf35094cc8d37542c9543cd16f3c62e7cd23df5775b410beaad850"} Jul 28 10:24:15.825340 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:15.825310 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" event={"ID":"08fb446f-5afe-4551-9e6a-5a21956bc677","Type":"ContainerStarted","Data":"7d31e8f5763f4f8d88d749f269419ec7eb42f8b545c2db6f96bc04aa2aca0237"} Jul 28 10:24:15.826267 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:15.826141 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:16.494932 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:16.494894 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:16.582820 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:16.582782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:16.710712 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:16.710673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:17.125062 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:17.125000 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:17.197205 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:17.197166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:20.839631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.839563 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xp886" event={"ID":"2d5d4cd7-0d96-471b-894c-fe57fd572b01","Type":"ContainerStarted","Data":"4d817ad6dc7b2143e89e675f1c48958586fbf9507d3e59a915f4044b74cb57a3"} Jul 28 10:24:20.841415 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.841376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nv8zj" event={"ID":"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9","Type":"ContainerStarted","Data":"5f61cf1d6427b492f167fadc4c03712874e090633bf2047bf53ec124cbb873eb"} Jul 28 10:24:20.841539 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.841437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nv8zj" event={"ID":"0ff2fb62-a7e2-4295-aceb-6fdeef1993b9","Type":"ContainerStarted","Data":"1a2e15bc8638d5ea6c91b0233f3339eafc7493dc5fece4ac08ded9f53879e97e"} Jul 28 10:24:20.841539 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.841465 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:20.842726 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.842700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2lsd5" event={"ID":"453b6262-344b-42ae-b448-629bce3a1a62","Type":"ContainerStarted","Data":"3e4cf5d397c410c62838ffeb9a28b9bacfdda08fcf64a6794b19185464a187c5"} Jul 28 10:24:20.854892 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.854845 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xp886" podStartSLOduration=1.308668931 podStartE2EDuration="6.854829094s" totalImagesPullingTime="5.546160163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:20.854504515 +0000 UTC m=+57.987163797" watchObservedRunningTime="2026-07-28 10:24:20.854829094 +0000 UTC m=+57.987488374" Jul 28 10:24:20.855313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.855289 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" podStartSLOduration=12.855274685 podStartE2EDuration="12.855274685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:15.847890464 +0000 UTC m=+52.980549748" watchObservedRunningTime="2026-07-28 10:24:20.855274685 +0000 UTC m=+57.987933964" Jul 28 10:24:20.873496 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:20.870915 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nv8zj" podStartSLOduration=1.3255694390000001 podStartE2EDuration="6.870898188s" totalImagesPullingTime="5.545328749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:20.870523539 +0000 UTC m=+58.003182819" watchObservedRunningTime="2026-07-28 10:24:20.870898188 +0000 UTC m=+58.003557468" Jul 28 10:24:22.846581 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:22.846548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:22.948204 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:22.948157 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:23.016503 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:23.016466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:24.775210 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:24.775180 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5gv9" Jul 28 10:24:24.855405 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:24.855361 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2lsd5" event={"ID":"453b6262-344b-42ae-b448-629bce3a1a62","Type":"ContainerStarted","Data":"705e53fadb53b47cec765142dfcc455b2c55b71395e52712dfc8124fa65b1bec"} Jul 28 10:24:24.928796 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:24.928756 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:24.982366 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:24.982335 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:25.037238 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:25.037142 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:25.650385 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:25.650306 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:26.051285 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:26.051242 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:26.158709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:26.158668 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:26.253026 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:26.252998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:26.353417 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:26.353284 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:26.724009 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:26.723965 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.136188 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.136099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.569431 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.569407 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.670084 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.670059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.767829 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.767774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.866505 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.866480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2lsd5" event={"ID":"453b6262-344b-42ae-b448-629bce3a1a62","Type":"ContainerStarted","Data":"562c18b86b018fdd2ec0a0825c6d511f8e5155e19b6fd3aabbc67b91b67ad6c1"} Jul 28 10:24:27.873296 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.873266 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:27.903106 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.903061 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2lsd5" podStartSLOduration=1.36048955 podStartE2EDuration="13.903048044s" totalImagesPullingTime="12.542558494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:27.901359666 +0000 UTC m=+65.034018946" watchObservedRunningTime="2026-07-28 10:24:27.903048044 +0000 UTC m=+65.035707323" Jul 28 10:24:27.915820 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.915794 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2"] Jul 28 10:24:27.918716 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.918701 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.923329 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.923307 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cntf4"] Jul 28 10:24:27.924228 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924204 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 10:24:27.924423 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924399 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 10:24:27.924423 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924419 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 10:24:27.924710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924560 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 10:24:27.924710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924581 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 10:24:27.924710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924616 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qbxq5\"" Jul 28 10:24:27.924942 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.924928 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 10:24:27.926121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.926103 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.932531 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.932516 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 10:24:27.932620 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.932552 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qpdwx\"" Jul 28 10:24:27.932679 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.932668 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 10:24:27.933519 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.933503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 10:24:27.941043 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.941023 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2"] Jul 28 10:24:27.981110 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlj8\" (UniqueName: \"kubernetes.io/projected/76063a07-970e-401b-ac66-c37b3c1d206b-kube-api-access-2hlj8\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981210 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981210 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981210 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-textfile\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-wtmp\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981260 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981289 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a87b2508-ae9d-47aa-a853-80eab2d5379e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981318 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-sys\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-metrics-client-ca\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-root\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmqmp\" (UniqueName: \"kubernetes.io/projected/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-api-access-pmqmp\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:27.981714 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981546 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:27.981714 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.981571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:27.984323 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.984307 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 10:24:27.994553 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:27.994529 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 10:24:28.006147 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.006099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvz8\" (UniqueName: \"kubernetes.io/projected/7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076-kube-api-access-5mvz8\") pod \"network-check-target-xnrgz\" (UID: \"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076\") " pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:28.013630 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.013610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.082826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082775 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.082826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082801 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.082826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-root\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmqmp\" (UniqueName: \"kubernetes.io/projected/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-api-access-pmqmp\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082900 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-root\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:28.082959 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.082990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:28.083004 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:24:28.083004 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls podName:76063a07-970e-401b-ac66-c37b3c1d206b nodeName:}" failed. No retries permitted until 2026-07-28 10:24:28.582989174 +0000 UTC m=+65.715648431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls") pod "node-exporter-cntf4" (UID: "76063a07-970e-401b-ac66-c37b3c1d206b") : secret "node-exporter-tls" not found Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlj8\" (UniqueName: \"kubernetes.io/projected/76063a07-970e-401b-ac66-c37b3c1d206b-kube-api-access-2hlj8\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083043 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-textfile\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-wtmp\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a87b2508-ae9d-47aa-a853-80eab2d5379e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-sys\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.083313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-metrics-client-ca\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083478 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-textfile\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-sys\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083856 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76063a07-970e-401b-ac66-c37b3c1d206b-metrics-client-ca\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.083925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.084297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.084115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-wtmp\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.084297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.084248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a87b2508-ae9d-47aa-a853-80eab2d5379e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.084396 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.084299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a87b2508-ae9d-47aa-a853-80eab2d5379e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.085414 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.085394 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.085684 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.085666 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.085892 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.085847 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.085892 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.085886 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 10:24:28.085982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.085967 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 10:24:28.093632 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.093612 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.095111 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.095089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e67fa4a7-0013-40fb-b97b-9ab58fa62120-original-pull-secret\") pod \"global-pull-secret-syncer-swstv\" (UID: \"e67fa4a7-0013-40fb-b97b-9ab58fa62120\") " pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:28.095326 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.095310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c-metrics-certs\") pod \"network-metrics-daemon-4st7m\" (UID: \"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c\") " pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:28.095784 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.095765 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmqmp\" (UniqueName: \"kubernetes.io/projected/a87b2508-ae9d-47aa-a853-80eab2d5379e-kube-api-access-pmqmp\") pod \"kube-state-metrics-56d9f944b4-h82g2\" (UID: \"a87b2508-ae9d-47aa-a853-80eab2d5379e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.105389 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.105372 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlj8\" (UniqueName: \"kubernetes.io/projected/76063a07-970e-401b-ac66-c37b3c1d206b-kube-api-access-2hlj8\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.192886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.192864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swstv" Jul 28 10:24:28.201872 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.201856 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mqqnj\"" Jul 28 10:24:28.206653 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.206636 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8src\"" Jul 28 10:24:28.208863 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.208847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:28.214719 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.214699 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4st7m" Jul 28 10:24:28.228640 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.228610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" Jul 28 10:24:28.255969 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.255921 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.355169 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.355144 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.355321 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.355233 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swstv"] Jul 28 10:24:28.359652 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:28.359629 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67fa4a7_0013_40fb_b97b_9ab58fa62120.slice/crio-3aab940c55875bc425f0753e4b9b2c7aabb1b3d2e1f0ff4f5889e1e91a06ef01 WatchSource:0}: Error finding container 3aab940c55875bc425f0753e4b9b2c7aabb1b3d2e1f0ff4f5889e1e91a06ef01: Status 404 returned error can't find the container with id 3aab940c55875bc425f0753e4b9b2c7aabb1b3d2e1f0ff4f5889e1e91a06ef01 Jul 28 10:24:28.374970 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:28.374940 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a32b5ac_0ce3_45e8_a29a_81f3fa3e4076.slice/crio-74d7826e1d1e75b40791558463c8fd085aee2d32e7dddce97a87d1b7ba8c7dfe WatchSource:0}: Error finding container 74d7826e1d1e75b40791558463c8fd085aee2d32e7dddce97a87d1b7ba8c7dfe: Status 404 returned error can't find the container with id 74d7826e1d1e75b40791558463c8fd085aee2d32e7dddce97a87d1b7ba8c7dfe Jul 28 10:24:28.377879 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.377855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xnrgz"] Jul 28 10:24:28.396876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.396854 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4st7m"] Jul 28 10:24:28.399574 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:28.399553 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f15c2b1_0ea0_4cef_8ff9_0ff9dbc1928c.slice/crio-68dda5d7518aaa1dd4ef7b60d4ef71f24fbd7ac9efaf0b174788b83302972c71 WatchSource:0}: Error finding container 68dda5d7518aaa1dd4ef7b60d4ef71f24fbd7ac9efaf0b174788b83302972c71: Status 404 returned error can't find the container with id 68dda5d7518aaa1dd4ef7b60d4ef71f24fbd7ac9efaf0b174788b83302972c71 Jul 28 10:24:28.417432 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.417410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2"] Jul 28 10:24:28.420263 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:28.420239 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87b2508_ae9d_47aa_a853_80eab2d5379e.slice/crio-f61507a83fde1e464e7dc6428334583371f21b9fcfa0b9c7641ded03c27a5690 WatchSource:0}: Error finding container f61507a83fde1e464e7dc6428334583371f21b9fcfa0b9c7641ded03c27a5690: Status 404 returned error can't find the container with id f61507a83fde1e464e7dc6428334583371f21b9fcfa0b9c7641ded03c27a5690 Jul 28 10:24:28.420957 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.420933 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.587154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.587086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.589447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.589425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76063a07-970e-401b-ac66-c37b3c1d206b-node-exporter-tls\") pod \"node-exporter-cntf4\" (UID: \"76063a07-970e-401b-ac66-c37b3c1d206b\") " pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.752637 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.752602 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:28.835126 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.835100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cntf4" Jul 28 10:24:28.860317 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:28.860284 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76063a07_970e_401b_ac66_c37b3c1d206b.slice/crio-c1449243ad992d9ef83145749791c9c12c1c0db29fe33ece494e0f76c276732b WatchSource:0}: Error finding container c1449243ad992d9ef83145749791c9c12c1c0db29fe33ece494e0f76c276732b: Status 404 returned error can't find the container with id c1449243ad992d9ef83145749791c9c12c1c0db29fe33ece494e0f76c276732b Jul 28 10:24:28.872137 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.872096 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cntf4" event={"ID":"76063a07-970e-401b-ac66-c37b3c1d206b","Type":"ContainerStarted","Data":"c1449243ad992d9ef83145749791c9c12c1c0db29fe33ece494e0f76c276732b"} Jul 28 10:24:28.873712 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.873686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" event={"ID":"a87b2508-ae9d-47aa-a853-80eab2d5379e","Type":"ContainerStarted","Data":"f61507a83fde1e464e7dc6428334583371f21b9fcfa0b9c7641ded03c27a5690"} Jul 28 10:24:28.875079 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.875055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4st7m" event={"ID":"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c","Type":"ContainerStarted","Data":"68dda5d7518aaa1dd4ef7b60d4ef71f24fbd7ac9efaf0b174788b83302972c71"} Jul 28 10:24:28.876423 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.876400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xnrgz" event={"ID":"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076","Type":"ContainerStarted","Data":"74d7826e1d1e75b40791558463c8fd085aee2d32e7dddce97a87d1b7ba8c7dfe"} Jul 28 10:24:28.877583 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.877561 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-swstv" event={"ID":"e67fa4a7-0013-40fb-b97b-9ab58fa62120","Type":"ContainerStarted","Data":"3aab940c55875bc425f0753e4b9b2c7aabb1b3d2e1f0ff4f5889e1e91a06ef01"} Jul 28 10:24:28.916747 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:28.914758 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:29.019918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.019883 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:29.144866 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.144782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:29.892504 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.891400 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64646cb4fb-fntvg"] Jul 28 10:24:29.908777 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.908175 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64646cb4fb-fntvg"] Jul 28 10:24:29.908777 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.908326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.911424 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.911399 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 10:24:29.911621 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.911595 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 10:24:29.911779 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.911698 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eq1hninebm9lf\"" Jul 28 10:24:29.912111 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.911843 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 10:24:29.912111 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.912036 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 10:24:29.912111 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.912056 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 10:24:29.912303 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.912160 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-smwg8\"" Jul 28 10:24:29.998188 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998257 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998289 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-metrics-client-ca\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn28j\" (UniqueName: \"kubernetes.io/projected/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-kube-api-access-nn28j\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998370 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:29.998626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:29.998395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-grpc-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099378 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-metrics-client-ca\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn28j\" (UniqueName: \"kubernetes.io/projected/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-kube-api-access-nn28j\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099470 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-grpc-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099682 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.099682 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.099551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.100821 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.100766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-metrics-client-ca\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.105195 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.105167 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.105964 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.105936 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.106136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.106090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.107867 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.107822 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.108386 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.108320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-grpc-tls\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.117241 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.117212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.125064 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.125040 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn28j\" (UniqueName: \"kubernetes.io/projected/9d4752ab-620c-4749-b5ea-bcc867ff6e8c-kube-api-access-nn28j\") pod \"thanos-querier-64646cb4fb-fntvg\" (UID: \"9d4752ab-620c-4749-b5ea-bcc867ff6e8c\") " pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.220245 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.220207 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:30.852675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.852004 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nv8zj" Jul 28 10:24:30.880849 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.880766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:30.895759 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.895536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" event={"ID":"a87b2508-ae9d-47aa-a853-80eab2d5379e","Type":"ContainerStarted","Data":"14deeac50dc4215404c64991b75b7a02f06c6dc8c80f3f4253cf851dd095da50"} Jul 28 10:24:30.895759 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.895607 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" event={"ID":"a87b2508-ae9d-47aa-a853-80eab2d5379e","Type":"ContainerStarted","Data":"355eaa94197dc15874cb3b1a89d9bb987bd3bc0d12285ca0764dde64f44cb363"} Jul 28 10:24:30.904267 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.904089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64646cb4fb-fntvg"] Jul 28 10:24:30.904525 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.904173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4st7m" event={"ID":"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c","Type":"ContainerStarted","Data":"39703e9abc009654cea6a7ee58e7497c6c40187afaaae30c796157b25fa711bc"} Jul 28 10:24:30.909672 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.908895 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cntf4" event={"ID":"76063a07-970e-401b-ac66-c37b3c1d206b","Type":"ContainerStarted","Data":"004e077a3c25e3ba63c7c921705157ac63095e15e6e5121a8f1a09bf98ef6b89"} Jul 28 10:24:30.946858 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:30.946490 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4752ab_620c_4749_b5ea_bcc867ff6e8c.slice/crio-deacf2bc16aaef139cbc9531c39dc37f7cc1c35a4199bcdb2342fad167e88979 WatchSource:0}: Error finding container deacf2bc16aaef139cbc9531c39dc37f7cc1c35a4199bcdb2342fad167e88979: Status 404 returned error can't find the container with id deacf2bc16aaef139cbc9531c39dc37f7cc1c35a4199bcdb2342fad167e88979 Jul 28 10:24:30.962997 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:30.962932 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:31.042202 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.042126 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:31.141465 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.141415 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:31.249962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.249898 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:31.913903 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.913866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"deacf2bc16aaef139cbc9531c39dc37f7cc1c35a4199bcdb2342fad167e88979"} Jul 28 10:24:31.915542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.915509 2567 generic.go:356] "Generic (PLEG): container finished" podID="76063a07-970e-401b-ac66-c37b3c1d206b" containerID="004e077a3c25e3ba63c7c921705157ac63095e15e6e5121a8f1a09bf98ef6b89" exitCode=0 Jul 28 10:24:31.915666 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.915582 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cntf4" event={"ID":"76063a07-970e-401b-ac66-c37b3c1d206b","Type":"ContainerDied","Data":"004e077a3c25e3ba63c7c921705157ac63095e15e6e5121a8f1a09bf98ef6b89"} Jul 28 10:24:31.917983 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.917953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" event={"ID":"a87b2508-ae9d-47aa-a853-80eab2d5379e","Type":"ContainerStarted","Data":"13a18579c8e6aafb7d9ae0a01f039693ca7741a687305057762580d022fc71f3"} Jul 28 10:24:31.919855 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.919831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4st7m" event={"ID":"8f15c2b1-0ea0-4cef-8ff9-0ff9dbc1928c","Type":"ContainerStarted","Data":"f1c80dab6ccac233799a4acc59bc552c53c3d6ca8d9b706c2005a28393607a99"} Jul 28 10:24:31.939869 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.939831 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h82g2" podStartSLOduration=2.702762171 podStartE2EDuration="4.939814023s" totalImagesPullingTime="2.237051852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:31.93844963 +0000 UTC m=+69.071108910" watchObservedRunningTime="2026-07-28 10:24:31.939814023 +0000 UTC m=+69.072473300" Jul 28 10:24:31.954840 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:31.954796 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4st7m" podStartSLOduration=66.699040947 podStartE2EDuration="1m8.954783458s" totalImagesPullingTime="2.255742511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:31.954194434 +0000 UTC m=+69.086853715" watchObservedRunningTime="2026-07-28 10:24:31.954783458 +0000 UTC m=+69.087442740" Jul 28 10:24:32.080921 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.080886 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:32.146065 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.146020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:32.415887 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.415852 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77797f6788-v5lvd"] Jul 28 10:24:32.419012 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.418988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.422964 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.422940 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 10:24:32.422964 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.422965 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d3e5ihtjsn5h1\"" Jul 28 10:24:32.423181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.423027 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x8qc4\"" Jul 28 10:24:32.423181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.423046 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 10:24:32.423181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.422966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 10:24:32.423181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.422966 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 10:24:32.432832 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.432808 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77797f6788-v5lvd"] Jul 28 10:24:32.520118 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-client-certs\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520260 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d30e430f-8bdf-4a42-94f8-915ea38d72c0-audit-log\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520260 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5sq\" (UniqueName: \"kubernetes.io/projected/d30e430f-8bdf-4a42-94f8-915ea38d72c0-kube-api-access-ct5sq\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520260 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-metrics-server-audit-profiles\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520260 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520441 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-tls\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.520441 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.520314 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-client-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.620898 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.620864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-tls\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.620905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-client-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.620950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-client-certs\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.620974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d30e430f-8bdf-4a42-94f8-915ea38d72c0-audit-log\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.620993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5sq\" (UniqueName: \"kubernetes.io/projected/d30e430f-8bdf-4a42-94f8-915ea38d72c0-kube-api-access-ct5sq\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621655 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.621313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-metrics-server-audit-profiles\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621655 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.621361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.621655 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.621603 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d30e430f-8bdf-4a42-94f8-915ea38d72c0-audit-log\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.622126 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.622103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.622264 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.622243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d30e430f-8bdf-4a42-94f8-915ea38d72c0-metrics-server-audit-profiles\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.624012 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.623988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-tls\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.624156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.624137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-secret-metrics-server-client-certs\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.625436 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.625415 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d30e430f-8bdf-4a42-94f8-915ea38d72c0-client-ca-bundle\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.632082 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.632056 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5sq\" (UniqueName: \"kubernetes.io/projected/d30e430f-8bdf-4a42-94f8-915ea38d72c0-kube-api-access-ct5sq\") pod \"metrics-server-77797f6788-v5lvd\" (UID: \"d30e430f-8bdf-4a42-94f8-915ea38d72c0\") " pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:32.633401 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.633376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:32.696595 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.696531 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:32.730456 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:32.730430 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:33.039423 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.039391 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:33.141321 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.141289 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:33.164874 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.164842 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b95489946-nl589"] Jul 28 10:24:33.168424 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.168406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.170962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.170936 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 10:24:33.171176 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.171161 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 10:24:33.171247 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.171204 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 10:24:33.171353 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.171336 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 10:24:33.171429 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.171407 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2lvgc\"" Jul 28 10:24:33.171569 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.171551 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 10:24:33.176565 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.176493 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 10:24:33.187549 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.187529 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b95489946-nl589"] Jul 28 10:24:33.208239 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.208204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:33.226187 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226279 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226211 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-serving-certs-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226279 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fbl\" (UniqueName: \"kubernetes.io/projected/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-kube-api-access-g8fbl\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-federate-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226502 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226502 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226449 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-metrics-client-ca\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.226502 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.226482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.326986 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.326909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.326986 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.326956 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-metrics-client-ca\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.326993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-serving-certs-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fbl\" (UniqueName: \"kubernetes.io/projected/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-kube-api-access-g8fbl\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-federate-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.327926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.327896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-metrics-client-ca\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.328258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.328214 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-serving-certs-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.328780 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.328717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.330332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.330305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.330442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.330390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-secret-telemeter-client\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.331078 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.331055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-telemeter-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.335106 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.335086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-federate-client-tls\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.337378 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.337346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fbl\" (UniqueName: \"kubernetes.io/projected/fc0c17b7-a505-4bf1-811a-bf63d9f9436f-kube-api-access-g8fbl\") pod \"telemeter-client-5b95489946-nl589\" (UID: \"fc0c17b7-a505-4bf1-811a-bf63d9f9436f\") " pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.479867 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.479838 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" Jul 28 10:24:33.680076 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.680051 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77797f6788-v5lvd"] Jul 28 10:24:33.710983 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.710902 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b95489946-nl589"] Jul 28 10:24:33.715794 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:33.715281 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0c17b7_a505_4bf1_811a_bf63d9f9436f.slice/crio-1c86964c3b57ddd8170a66019b7c555513dd78333b150944f7b0de531b6ca1cb WatchSource:0}: Error finding container 1c86964c3b57ddd8170a66019b7c555513dd78333b150944f7b0de531b6ca1cb: Status 404 returned error can't find the container with id 1c86964c3b57ddd8170a66019b7c555513dd78333b150944f7b0de531b6ca1cb Jul 28 10:24:33.933328 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.933245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cntf4" event={"ID":"76063a07-970e-401b-ac66-c37b3c1d206b","Type":"ContainerStarted","Data":"5c939c201b269bf4a8c4a569e7bd0026ae1a3901433dd177035559278228535f"} Jul 28 10:24:33.933328 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.933309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cntf4" event={"ID":"76063a07-970e-401b-ac66-c37b3c1d206b","Type":"ContainerStarted","Data":"ec96bb5654fc3ae5548d263a9c513995285935d0af521b9b5f78e85e38dfae6e"} Jul 28 10:24:33.934710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.934667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" event={"ID":"fc0c17b7-a505-4bf1-811a-bf63d9f9436f","Type":"ContainerStarted","Data":"1c86964c3b57ddd8170a66019b7c555513dd78333b150944f7b0de531b6ca1cb"} Jul 28 10:24:33.936011 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.935986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" event={"ID":"d30e430f-8bdf-4a42-94f8-915ea38d72c0","Type":"ContainerStarted","Data":"16f19ec2419568deec78cba33ca187ba92b70e208a132b5d81b64652049d5628"} Jul 28 10:24:33.937594 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.937569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xnrgz" event={"ID":"7a32b5ac-0ce3-45e8-a29a-81f3fa3e4076","Type":"ContainerStarted","Data":"11794bad2d0540a8bf125fa09965bcc5827917454f40eafa27802ed980e31ddd"} Jul 28 10:24:33.937721 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.937626 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:24:33.939425 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.939403 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-swstv" event={"ID":"e67fa4a7-0013-40fb-b97b-9ab58fa62120","Type":"ContainerStarted","Data":"f9bdd377eea8d4412fdee99c79a11236e621c64e5c678aa342699ddfffd1ab43"} Jul 28 10:24:33.952019 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.951971 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cntf4" podStartSLOduration=5.034760839 podStartE2EDuration="6.951954192s" totalImagesPullingTime="1.798860213s" totalInitContainerRuntime="118.33314ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:33.951214309 +0000 UTC m=+71.083873589" watchObservedRunningTime="2026-07-28 10:24:33.951954192 +0000 UTC m=+71.084613473" Jul 28 10:24:33.969886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.969846 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-swstv" podStartSLOduration=65.795498798 podStartE2EDuration="1m10.969834793s" totalImagesPullingTime="5.174335995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:33.969469106 +0000 UTC m=+71.102128390" watchObservedRunningTime="2026-07-28 10:24:33.969834793 +0000 UTC m=+71.102494051" Jul 28 10:24:33.988837 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:33.988795 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xnrgz" podStartSLOduration=65.841364099 podStartE2EDuration="1m10.988783684s" totalImagesPullingTime="5.147419585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:33.988122986 +0000 UTC m=+71.120782265" watchObservedRunningTime="2026-07-28 10:24:33.988783684 +0000 UTC m=+71.121442963" Jul 28 10:24:34.216516 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:34.216486 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:34.436243 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:34.436207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:34.535330 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:34.535256 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:36.836406 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.836366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:36.954709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.954669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"c1aaa1d504c49f504944573ae7dc876b0c6d92453e978ad443bbd57d41eba29c"} Jul 28 10:24:36.954878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.954772 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"7906bed674c4449fa2ab8257f4e2e4001339448a075168cbc6830532f1ef517f"} Jul 28 10:24:36.954878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.954824 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"71c98c6f90c16fbdff5f9fd97e2993284e57e612342f1ab5be92730d0d73ea4f"} Jul 28 10:24:36.956322 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.956295 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" event={"ID":"fc0c17b7-a505-4bf1-811a-bf63d9f9436f","Type":"ContainerStarted","Data":"615d4643dc4219dd23d56f64259e5de23193a950ae91fbd1d3b2316268d89ec7"} Jul 28 10:24:36.957836 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.957811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" event={"ID":"d30e430f-8bdf-4a42-94f8-915ea38d72c0","Type":"ContainerStarted","Data":"deddbd14ec2066d8f4f2ffbcf25e22203e61cb552b28080e02d0b63c94e9efb8"} Jul 28 10:24:36.978297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:36.978250 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" podStartSLOduration=2.542884559 podStartE2EDuration="4.978231812s" totalImagesPullingTime="2.435347253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:36.977778535 +0000 UTC m=+74.110437830" watchObservedRunningTime="2026-07-28 10:24:36.978231812 +0000 UTC m=+74.110891092" Jul 28 10:24:37.835152 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.835128 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5879f5564b-9vffh" Jul 28 10:24:37.964338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.964301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"8d7d8edffc7fa2f076729fad8771295a7f4883301d31844cb73a8df1f9a2a917"} Jul 28 10:24:37.964757 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.964382 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"e7d485581cb14fa5dd630bc142c44f13d95369faa2fceb6797f5e954f9d18d02"} Jul 28 10:24:37.964757 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.964432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" event={"ID":"9d4752ab-620c-4749-b5ea-bcc867ff6e8c","Type":"ContainerStarted","Data":"f5b6701fac673d34aa137fd3fedb4b60e634a9259ad34bd29aca5d75c06a960f"} Jul 28 10:24:37.964757 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.964474 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:37.966278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.966256 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" event={"ID":"fc0c17b7-a505-4bf1-811a-bf63d9f9436f","Type":"ContainerStarted","Data":"216d3228bd5cf6bfec42e7d594fe5e69030c6f82daca99f42960730dfc88467b"} Jul 28 10:24:37.966397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.966308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" event={"ID":"fc0c17b7-a505-4bf1-811a-bf63d9f9436f","Type":"ContainerStarted","Data":"b16fe88e040575d99145314c3d40e4b5b0acb6faed7687face6305e29ea7bec3"} Jul 28 10:24:37.989790 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:37.989715 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" podStartSLOduration=2.783973771 podStartE2EDuration="8.989700975s" totalImagesPullingTime="6.205727204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:37.989028437 +0000 UTC m=+75.121687732" watchObservedRunningTime="2026-07-28 10:24:37.989700975 +0000 UTC m=+75.122360289" Jul 28 10:24:38.020130 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:38.020102 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:38.021082 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:38.021041 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b95489946-nl589" podStartSLOduration=1.45936644 podStartE2EDuration="5.021021297s" totalImagesPullingTime="3.561654857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:38.018605455 +0000 UTC m=+75.151264724" watchObservedRunningTime="2026-07-28 10:24:38.021021297 +0000 UTC m=+75.153680578" Jul 28 10:24:38.115981 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:38.115916 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:38.424716 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:38.424644 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:38.517622 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:38.517593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:40.045444 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:40.045407 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:40.150700 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:40.150670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:41.552229 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:41.552195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:43.182581 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.182541 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:24:43.187128 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.187106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.191866 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.191841 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 10:24:43.191966 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.191842 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 10:24:43.191966 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.191845 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 10:24:43.192962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.192944 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5kjh\"" Jul 28 10:24:43.193045 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.192978 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 10:24:43.193263 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.193188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 10:24:43.193263 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.193234 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 10:24:43.193389 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.193366 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 10:24:43.197949 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.197929 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 10:24:43.200781 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.200761 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:24:43.303474 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303450 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303792 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303615 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf52p\" (UniqueName: \"kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303792 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.303792 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.303706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.404903 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.404883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405014 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.404910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405014 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.404939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405320 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf52p\" (UniqueName: \"kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405672 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405646 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405777 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405646 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.405991 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.405970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.406034 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.406002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.407447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.407423 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.407711 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.407693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.414148 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.414129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf52p\" (UniqueName: \"kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p\") pod \"console-7bbc757f44-rdmjz\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.498140 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.498119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:43.504100 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.504079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:43.606197 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.606164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:43.655474 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.655438 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:24:43.658065 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:24:43.658040 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e60cc9a_5093_4303_a9b5_bf1c50e6a3c2.slice/crio-97b4d4454e9048293e16483b3cec2f462e899ed11a3e4317b8fae6803ca55cda WatchSource:0}: Error finding container 97b4d4454e9048293e16483b3cec2f462e899ed11a3e4317b8fae6803ca55cda: Status 404 returned error can't find the container with id 97b4d4454e9048293e16483b3cec2f462e899ed11a3e4317b8fae6803ca55cda Jul 28 10:24:43.977219 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.977113 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64646cb4fb-fntvg" Jul 28 10:24:43.989707 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:43.989680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbc757f44-rdmjz" event={"ID":"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2","Type":"ContainerStarted","Data":"97b4d4454e9048293e16483b3cec2f462e899ed11a3e4317b8fae6803ca55cda"} Jul 28 10:24:46.363195 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:46.363149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:46.420371 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:46.420347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:47.003144 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:47.003099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbc757f44-rdmjz" event={"ID":"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2","Type":"ContainerStarted","Data":"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01"} Jul 28 10:24:47.027063 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:47.027007 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bbc757f44-rdmjz" podStartSLOduration=1.215511374 podStartE2EDuration="4.026990477s" totalImagesPullingTime="2.811479103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:47.025967318 +0000 UTC m=+84.158626598" watchObservedRunningTime="2026-07-28 10:24:47.026990477 +0000 UTC m=+84.159649756" Jul 28 10:24:49.060489 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:49.060456 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:49.162038 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:49.161999 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:51.403759 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:51.403693 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:51.499778 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:51.499730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:51.571778 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:51.571755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:52.730636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:52.730605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:52.731019 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:52.730659 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:24:53.421379 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:53.421342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:53.496327 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:53.496301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:53.499234 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:53.499216 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:53.499289 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:53.499278 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:53.504066 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:53.504047 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:54.032919 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:54.032896 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:24:57.584538 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:57.584506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:57.656392 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:57.656364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:58.364045 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:58.364011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:58.424936 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:58.424904 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:58.966285 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:58.966252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:24:59.034912 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:24:59.034877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:04.946436 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:04.946406 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xnrgz" Jul 28 10:25:08.418441 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:08.418404 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:08.474535 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:08.474500 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:12.736263 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:12.736229 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:25:12.740056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:12.740033 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77797f6788-v5lvd" Jul 28 10:25:43.978906 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:43.978863 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:44.040902 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:44.040867 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:48.815974 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.815940 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:25:48.819261 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.819241 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.830761 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.830705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:25:48.905075 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.905046 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:48.959348 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959442 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959410 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdwm\" (UniqueName: \"kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.959626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.959527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:48.967864 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:48.967831 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:49.060886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdwm\" (UniqueName: \"kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.060995 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.060994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.061761 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.061710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.061856 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.061710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.061856 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.061723 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.062178 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.062156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.063480 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.063454 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.063557 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.063544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.070856 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.070810 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdwm\" (UniqueName: \"kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm\") pod \"console-64bb4c968c-tzndc\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.129948 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.129928 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:49.263215 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:49.263184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:25:49.265190 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:25:49.265157 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f31a924_f357_42a0_90bd_edfec619ba37.slice/crio-d26a09fdbe37189aeee2987ada8c9abc95a8ee2754514c10cb0faf8587cdf5c2 WatchSource:0}: Error finding container d26a09fdbe37189aeee2987ada8c9abc95a8ee2754514c10cb0faf8587cdf5c2: Status 404 returned error can't find the container with id d26a09fdbe37189aeee2987ada8c9abc95a8ee2754514c10cb0faf8587cdf5c2 Jul 28 10:25:50.233994 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.233961 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb4c968c-tzndc" event={"ID":"7f31a924-f357-42a0-90bd-edfec619ba37","Type":"ContainerStarted","Data":"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9"} Jul 28 10:25:50.234376 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.234013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb4c968c-tzndc" event={"ID":"7f31a924-f357-42a0-90bd-edfec619ba37","Type":"ContainerStarted","Data":"d26a09fdbe37189aeee2987ada8c9abc95a8ee2754514c10cb0faf8587cdf5c2"} Jul 28 10:25:50.250437 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.250361 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64bb4c968c-tzndc" podStartSLOduration=2.250343056 podStartE2EDuration="2.250343056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:50.250180578 +0000 UTC m=+147.382839858" watchObservedRunningTime="2026-07-28 10:25:50.250343056 +0000 UTC m=+147.383002336" Jul 28 10:25:50.563435 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.563361 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 10:25:50.567113 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.567093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.569798 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.569772 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 10:25:50.569918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.569809 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 10:25:50.569918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.569781 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 10:25:50.569918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.569886 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fwhw9\"" Jul 28 10:25:50.570056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.569926 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 10:25:50.570152 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.570136 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 10:25:50.570438 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.570408 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 10:25:50.570438 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.570433 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 10:25:50.570581 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.570412 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 10:25:50.575518 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.575501 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 10:25:50.580850 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.580829 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 10:25:50.675368 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675348 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-out\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675590 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675487 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675590 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675590 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-web-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675590 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675603 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675622 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dw9v\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-kube-api-access-8dw9v\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.675709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.675693 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776307 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776398 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776398 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776398 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-web-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776398 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776676 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776730 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776803 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dw9v\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-kube-api-access-8dw9v\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776858 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776914 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.776914 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.777009 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.776924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-out\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.777058 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.777022 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.777644 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.777616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.779266 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.778921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.779764 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.779693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780359 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.779864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-web-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780448 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.780375 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780448 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.780430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.780649 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780860 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.780839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.780975 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.780957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.782130 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.782114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-config-out\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.782223 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.782205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.785051 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.785033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dw9v\" (UniqueName: \"kubernetes.io/projected/2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f-kube-api-access-8dw9v\") pod \"alertmanager-main-0\" (UID: \"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:50.877626 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:50.877573 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 10:25:51.012621 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:51.012596 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 10:25:51.013958 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:25:51.013925 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8cdfd6_c6c2_4772_949b_0dcfc785cc2f.slice/crio-7bd43d02d22b7c2039b770bdaccb4414fab2e8c9ea878ced0ea71491bcb96b46 WatchSource:0}: Error finding container 7bd43d02d22b7c2039b770bdaccb4414fab2e8c9ea878ced0ea71491bcb96b46: Status 404 returned error can't find the container with id 7bd43d02d22b7c2039b770bdaccb4414fab2e8c9ea878ced0ea71491bcb96b46 Jul 28 10:25:51.239172 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:51.239140 2567 generic.go:356] "Generic (PLEG): container finished" podID="2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f" containerID="387b400f9474ade1f672e3bb7ecd7363cea8fd200f1d6b5d99764c950661f22c" exitCode=0 Jul 28 10:25:51.239498 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:51.239230 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerDied","Data":"387b400f9474ade1f672e3bb7ecd7363cea8fd200f1d6b5d99764c950661f22c"} Jul 28 10:25:51.239498 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:51.239340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"7bd43d02d22b7c2039b770bdaccb4414fab2e8c9ea878ced0ea71491bcb96b46"} Jul 28 10:25:53.250720 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"5553f674e87b0ba5178109e00d2c0f287c39a0304d1728502b3e1e82c0c3ecdb"} Jul 28 10:25:53.251136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"c6f3c38a107efac610c1fa5eb51c6e3e14cf3705827c8dca562242605210fd2f"} Jul 28 10:25:53.251136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250806 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"b54d9b16249a9d66458c3e6e3b8bf59d284d05482b8274f64251367aa92db973"} Jul 28 10:25:53.251136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250830 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"64c12014e69cf1ceebde221bec63cbb46a903134c50dee609bc8372d5789447e"} Jul 28 10:25:53.251136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"a1ad7141e7e879547cd2d90e33bd1ab63c44d3119dba7f5cf7eed5099351fda8"} Jul 28 10:25:53.251136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.250880 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f","Type":"ContainerStarted","Data":"2e2040ca3e35f887a3d259b55ab90209e0f73d72215a883ce696c70623ec09e4"} Jul 28 10:25:53.280599 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:53.280555 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.718256242 podStartE2EDuration="3.280539333s" totalImagesPullingTime="1.55182438s" totalInitContainerRuntime="10.458711ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:53.27843887 +0000 UTC m=+150.411098166" watchObservedRunningTime="2026-07-28 10:25:53.280539333 +0000 UTC m=+150.413198611" Jul 28 10:25:55.881475 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:55.881429 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:55.945832 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:55.945799 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:59.131002 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.130964 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:59.131509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.131067 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:59.135823 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.135797 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:59.277213 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.277188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:25:59.327192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.327154 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:25:59.432326 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.432252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:59.487792 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.487763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:25:59.560130 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:25:59.560108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:08.419516 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:08.419480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:08.520221 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:08.520188 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:13.662619 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:13.662581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:24.346678 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.346625 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bbc757f44-rdmjz" podUID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" containerName="console" containerID="cri-o://573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01" gracePeriod=15 Jul 28 10:26:24.585752 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.585703 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:26:24.620831 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.620768 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf52p\" (UniqueName: \"kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.620831 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.620821 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.620857 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.620901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.620972 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621016 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621213 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621072 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert\") pod \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\" (UID: \"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2\") " Jul 28 10:26:24.621281 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621259 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:26:24.621281 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621268 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config" (OuterVolumeSpecName: "console-config") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:26:24.621383 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621354 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.621383 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621368 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.621613 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621584 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:26:24.621934 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.621902 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:26:24.623372 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.623336 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:26:24.623372 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.623358 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:26:24.623783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.623721 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p" (OuterVolumeSpecName: "kube-api-access-pf52p") pod "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" (UID: "4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2"). InnerVolumeSpecName "kube-api-access-pf52p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:26:24.721943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.721920 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.721943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.721943 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-trusted-ca-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.722093 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.721957 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-console-oauth-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.722093 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.721968 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-oauth-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:24.722093 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:24.721980 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf52p\" (UniqueName: \"kubernetes.io/projected/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2-kube-api-access-pf52p\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:26:25.192797 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.192755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:25.245552 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.245516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:26:25.371918 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.371888 2567 generic.go:356] "Generic (PLEG): container finished" podID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" containerID="573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01" exitCode=0 Jul 28 10:26:25.372258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.371942 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbc757f44-rdmjz" Jul 28 10:26:25.372258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.371954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbc757f44-rdmjz" event={"ID":"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2","Type":"ContainerDied","Data":"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01"} Jul 28 10:26:25.372258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.372004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbc757f44-rdmjz" event={"ID":"4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2","Type":"ContainerDied","Data":"97b4d4454e9048293e16483b3cec2f462e899ed11a3e4317b8fae6803ca55cda"} Jul 28 10:26:25.372258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.372032 2567 scope.go:117] "RemoveContainer" containerID="573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01" Jul 28 10:26:25.381006 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.380985 2567 scope.go:117] "RemoveContainer" containerID="573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01" Jul 28 10:26:25.381266 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:26:25.381247 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01\": container with ID starting with 573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01 not found: ID does not exist" containerID="573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01" Jul 28 10:26:25.381331 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.381273 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01"} err="failed to get container status \"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01\": rpc error: code = NotFound desc = could not find container \"573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01\": container with ID starting with 573f01b8ffb94caea6de378d9bb9c06bb5bce1383d3db9c361b312c03a7c5e01 not found: ID does not exist" Jul 28 10:26:25.393372 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.393350 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:26:25.396691 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:25.396671 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bbc757f44-rdmjz"] Jul 28 10:26:27.384903 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:27.384870 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" path="/var/lib/kubelet/pods/4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2/volumes" Jul 28 10:26:59.294774 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:26:59.294717 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:28:18.684658 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:28:18.684621 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:28:23.189835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:28:23.189810 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 10:28:56.802641 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:28:56.802602 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:28:56.858207 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:28:56.858175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:29:44.587383 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.587342 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7456544f99-mzlcx"] Jul 28 10:29:44.587982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.587778 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" containerName="console" Jul 28 10:29:44.587982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.587797 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" containerName="console" Jul 28 10:29:44.587982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.587891 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e60cc9a-5093-4303-a9b5-bf1c50e6a3c2" containerName="console" Jul 28 10:29:44.590685 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.590655 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.628672 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.628649 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7456544f99-mzlcx"] Jul 28 10:29:44.692528 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.692495 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:29:44.699399 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-service-ca\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-trusted-ca-bundle\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-oauth-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699466 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmsh\" (UniqueName: \"kubernetes.io/projected/04ddf026-8ed6-4190-afaf-cedc08a77bcf-kube-api-access-skmsh\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699592 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699592 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-oauth-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.699655 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.699589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.749510 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.749487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:29:44.800552 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-service-ca\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800554 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-trusted-ca-bundle\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-oauth-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmsh\" (UniqueName: \"kubernetes.io/projected/04ddf026-8ed6-4190-afaf-cedc08a77bcf-kube-api-access-skmsh\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-oauth-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.800930 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.800823 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.801278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.801253 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-service-ca\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.801407 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.801378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-oauth-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.801686 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.801384 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.801772 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.801722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04ddf026-8ed6-4190-afaf-cedc08a77bcf-trusted-ca-bundle\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.803218 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.803199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-serving-cert\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.803302 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.803285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04ddf026-8ed6-4190-afaf-cedc08a77bcf-console-oauth-config\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.809660 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.809634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmsh\" (UniqueName: \"kubernetes.io/projected/04ddf026-8ed6-4190-afaf-cedc08a77bcf-kube-api-access-skmsh\") pod \"console-7456544f99-mzlcx\" (UID: \"04ddf026-8ed6-4190-afaf-cedc08a77bcf\") " pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:44.899559 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:44.899505 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:45.029541 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:45.029513 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7456544f99-mzlcx"] Jul 28 10:29:45.031550 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:29:45.031525 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ddf026_8ed6_4190_afaf_cedc08a77bcf.slice/crio-cde37d760a0ac7c4311a91fabc5d6b5343825794a49311ac1221e9acd87308e4 WatchSource:0}: Error finding container cde37d760a0ac7c4311a91fabc5d6b5343825794a49311ac1221e9acd87308e4: Status 404 returned error can't find the container with id cde37d760a0ac7c4311a91fabc5d6b5343825794a49311ac1221e9acd87308e4 Jul 28 10:29:45.121730 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:45.121695 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7456544f99-mzlcx" event={"ID":"04ddf026-8ed6-4190-afaf-cedc08a77bcf","Type":"ContainerStarted","Data":"c6b36595a7191fe826f2424ac1ed168439a051fd58e432f1d826f23e222ef523"} Jul 28 10:29:45.121855 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:45.121768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7456544f99-mzlcx" event={"ID":"04ddf026-8ed6-4190-afaf-cedc08a77bcf","Type":"ContainerStarted","Data":"cde37d760a0ac7c4311a91fabc5d6b5343825794a49311ac1221e9acd87308e4"} Jul 28 10:29:45.141143 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:45.141102 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7456544f99-mzlcx" podStartSLOduration=1.141086389 podStartE2EDuration="1.141086389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:45.140702107 +0000 UTC m=+382.273361387" watchObservedRunningTime="2026-07-28 10:29:45.141086389 +0000 UTC m=+382.273745667" Jul 28 10:29:54.900646 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:54.900595 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:54.901032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:54.900678 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:54.905510 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:54.905489 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:55.163936 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:55.163859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7456544f99-mzlcx" Jul 28 10:29:55.209364 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:55.209337 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:29:55.312527 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:55.312492 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:29:55.412275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:55.412235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:29:55.512945 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:29:55.512908 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:30:20.228648 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.228610 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64bb4c968c-tzndc" podUID="7f31a924-f357-42a0-90bd-edfec619ba37" containerName="console" containerID="cri-o://1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9" gracePeriod=15 Jul 28 10:30:20.475921 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.475895 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:30:20.553016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.552947 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553016 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.552993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553196 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553025 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553196 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553079 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553196 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553157 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvdwm\" (UniqueName: \"kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553341 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553196 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553341 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553219 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config\") pod \"7f31a924-f357-42a0-90bd-edfec619ba37\" (UID: \"7f31a924-f357-42a0-90bd-edfec619ba37\") " Jul 28 10:30:20.553447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553321 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config" (OuterVolumeSpecName: "console-config") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:30:20.553528 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553504 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:30:20.553528 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553509 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:30:20.553654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.553625 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:30:20.555403 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.555375 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm" (OuterVolumeSpecName: "kube-api-access-tvdwm") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "kube-api-access-tvdwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:30:20.555526 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.555512 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:30:20.555595 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.555566 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f31a924-f357-42a0-90bd-edfec619ba37" (UID: "7f31a924-f357-42a0-90bd-edfec619ba37"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:30:20.654433 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654411 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654433 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654430 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdwm\" (UniqueName: \"kubernetes.io/projected/7f31a924-f357-42a0-90bd-edfec619ba37-kube-api-access-tvdwm\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654441 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-trusted-ca-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654450 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-oauth-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654460 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-console-config\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654468 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f31a924-f357-42a0-90bd-edfec619ba37-console-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.654567 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.654476 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f31a924-f357-42a0-90bd-edfec619ba37-oauth-serving-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:30:20.776136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:20.776103 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:30:21.255749 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.255712 2567 generic.go:356] "Generic (PLEG): container finished" podID="7f31a924-f357-42a0-90bd-edfec619ba37" containerID="1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9" exitCode=0 Jul 28 10:30:21.256132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.255781 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb4c968c-tzndc" Jul 28 10:30:21.256132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.255796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb4c968c-tzndc" event={"ID":"7f31a924-f357-42a0-90bd-edfec619ba37","Type":"ContainerDied","Data":"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9"} Jul 28 10:30:21.256132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.255850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb4c968c-tzndc" event={"ID":"7f31a924-f357-42a0-90bd-edfec619ba37","Type":"ContainerDied","Data":"d26a09fdbe37189aeee2987ada8c9abc95a8ee2754514c10cb0faf8587cdf5c2"} Jul 28 10:30:21.256132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.255873 2567 scope.go:117] "RemoveContainer" containerID="1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9" Jul 28 10:30:21.267167 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.267130 2567 scope.go:117] "RemoveContainer" containerID="1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9" Jul 28 10:30:21.267427 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:30:21.267406 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9\": container with ID starting with 1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9 not found: ID does not exist" containerID="1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9" Jul 28 10:30:21.267519 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.267438 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9"} err="failed to get container status \"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9\": rpc error: code = NotFound desc = could not find container \"1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9\": container with ID starting with 1dde38850b280c1adf589cdee2bb8cc21bbeea1396db9ef96128edc7d2abcdd9 not found: ID does not exist" Jul 28 10:30:21.279530 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.279508 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:30:21.283028 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.283005 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64bb4c968c-tzndc"] Jul 28 10:30:21.385287 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:30:21.385263 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f31a924-f357-42a0-90bd-edfec619ba37" path="/var/lib/kubelet/pods/7f31a924-f357-42a0-90bd-edfec619ba37/volumes" Jul 28 10:31:48.335625 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:31:48.335586 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:33:56.590013 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:56.589912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:33:57.230257 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.230224 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc"] Jul 28 10:33:57.230520 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.230508 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f31a924-f357-42a0-90bd-edfec619ba37" containerName="console" Jul 28 10:33:57.230563 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.230522 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f31a924-f357-42a0-90bd-edfec619ba37" containerName="console" Jul 28 10:33:57.230601 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.230576 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f31a924-f357-42a0-90bd-edfec619ba37" containerName="console" Jul 28 10:33:57.233453 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.233436 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.236177 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.236148 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 10:33:57.236298 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.236155 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 10:33:57.236298 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.236250 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knqhc\"" Jul 28 10:33:57.247518 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.247496 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc"] Jul 28 10:33:57.341798 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.341775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.341902 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.341811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtnw\" (UniqueName: \"kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.341902 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.341860 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.442944 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.442923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.443043 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.442974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.443137 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.443106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtnw\" (UniqueName: \"kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.443279 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.443262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.443325 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.443307 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.452627 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.452597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtnw\" (UniqueName: \"kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.543078 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.543028 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:33:57.668847 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.668818 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc"] Jul 28 10:33:57.671722 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:33:57.671697 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f33899_fcb0_4847_a596_d453f46f575e.slice/crio-3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b WatchSource:0}: Error finding container 3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b: Status 404 returned error can't find the container with id 3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b Jul 28 10:33:57.673546 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.673530 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:33:57.759919 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:57.759890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:33:58.049834 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:33:58.049798 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerStarted","Data":"3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b"} Jul 28 10:34:04.076510 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:04.076467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerStarted","Data":"22bc0260fa11b73b85eea51b30e74348c9335d9e1aadc07cbf978c21c1c923fd"} Jul 28 10:34:05.082014 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:05.081975 2567 generic.go:356] "Generic (PLEG): container finished" podID="85f33899-fcb0-4847-a596-d453f46f575e" containerID="22bc0260fa11b73b85eea51b30e74348c9335d9e1aadc07cbf978c21c1c923fd" exitCode=0 Jul 28 10:34:05.082396 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:05.082063 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerDied","Data":"22bc0260fa11b73b85eea51b30e74348c9335d9e1aadc07cbf978c21c1c923fd"} Jul 28 10:34:08.095655 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:08.095614 2567 generic.go:356] "Generic (PLEG): container finished" podID="85f33899-fcb0-4847-a596-d453f46f575e" containerID="00fc95cdc2fc32220c00cf9220233a567a6f7a942fbd25521e9419eb08d57b21" exitCode=0 Jul 28 10:34:08.096086 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:08.095704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerDied","Data":"00fc95cdc2fc32220c00cf9220233a567a6f7a942fbd25521e9419eb08d57b21"} Jul 28 10:34:15.127445 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:15.127405 2567 generic.go:356] "Generic (PLEG): container finished" podID="85f33899-fcb0-4847-a596-d453f46f575e" containerID="8f8a84d3b00d774022ad194843cae6a9e3acb36a727fd54f04fd7d2652fbe4fe" exitCode=0 Jul 28 10:34:15.127831 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:15.127497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerDied","Data":"8f8a84d3b00d774022ad194843cae6a9e3acb36a727fd54f04fd7d2652fbe4fe"} Jul 28 10:34:16.257151 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.257127 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:34:16.291869 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.291845 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util\") pod \"85f33899-fcb0-4847-a596-d453f46f575e\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " Jul 28 10:34:16.291999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.291885 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjtnw\" (UniqueName: \"kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw\") pod \"85f33899-fcb0-4847-a596-d453f46f575e\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " Jul 28 10:34:16.291999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.291925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle\") pod \"85f33899-fcb0-4847-a596-d453f46f575e\" (UID: \"85f33899-fcb0-4847-a596-d453f46f575e\") " Jul 28 10:34:16.292569 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.292544 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle" (OuterVolumeSpecName: "bundle") pod "85f33899-fcb0-4847-a596-d453f46f575e" (UID: "85f33899-fcb0-4847-a596-d453f46f575e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:34:16.294148 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.294117 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw" (OuterVolumeSpecName: "kube-api-access-pjtnw") pod "85f33899-fcb0-4847-a596-d453f46f575e" (UID: "85f33899-fcb0-4847-a596-d453f46f575e"). InnerVolumeSpecName "kube-api-access-pjtnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:34:16.296236 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.296217 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util" (OuterVolumeSpecName: "util") pod "85f33899-fcb0-4847-a596-d453f46f575e" (UID: "85f33899-fcb0-4847-a596-d453f46f575e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:34:16.393157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.393101 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:16.393157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.393133 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjtnw\" (UniqueName: \"kubernetes.io/projected/85f33899-fcb0-4847-a596-d453f46f575e-kube-api-access-pjtnw\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:16.393157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:16.393144 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85f33899-fcb0-4847-a596-d453f46f575e-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:17.137935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:17.137898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" event={"ID":"85f33899-fcb0-4847-a596-d453f46f575e","Type":"ContainerDied","Data":"3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b"} Jul 28 10:34:17.137935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:17.137921 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hx5fc" Jul 28 10:34:17.138147 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:17.137949 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4384c0fb1565df14a12c9535138693ad7b4684ab81bc955a02a64db6deaa8b" Jul 28 10:34:28.408032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:28.407996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:30.208170 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.208130 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:30.450790 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.450755 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7"] Jul 28 10:34:30.451107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451090 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="pull" Jul 28 10:34:30.451107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451108 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="pull" Jul 28 10:34:30.451191 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451129 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="extract" Jul 28 10:34:30.451191 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451135 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="extract" Jul 28 10:34:30.451191 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451144 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="util" Jul 28 10:34:30.451191 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451150 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="util" Jul 28 10:34:30.451327 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.451217 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f33899-fcb0-4847-a596-d453f46f575e" containerName="extract" Jul 28 10:34:30.454494 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.454478 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.457342 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.457311 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 10:34:30.457342 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.457323 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-n69ss\"" Jul 28 10:34:30.457342 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.457323 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:34:30.468872 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.468848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7"] Jul 28 10:34:30.593018 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.592991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjk5\" (UniqueName: \"kubernetes.io/projected/0155405c-6cda-4869-b9da-7ba7d508058d-kube-api-access-sjjk5\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.593018 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.593020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0155405c-6cda-4869-b9da-7ba7d508058d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.694257 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.694230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjk5\" (UniqueName: \"kubernetes.io/projected/0155405c-6cda-4869-b9da-7ba7d508058d-kube-api-access-sjjk5\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.694257 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.694257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0155405c-6cda-4869-b9da-7ba7d508058d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.694586 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.694570 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0155405c-6cda-4869-b9da-7ba7d508058d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.702888 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.702860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjk5\" (UniqueName: \"kubernetes.io/projected/0155405c-6cda-4869-b9da-7ba7d508058d-kube-api-access-sjjk5\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vbc7\" (UID: \"0155405c-6cda-4869-b9da-7ba7d508058d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.763922 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.763870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" Jul 28 10:34:30.910844 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:30.910817 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7"] Jul 28 10:34:30.912405 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:30.912374 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0155405c_6cda_4869_b9da_7ba7d508058d.slice/crio-db4edf97529e9e0d6ab7a12b14a64ead2ae9b7993b1f1513e5c50a95bebbc77c WatchSource:0}: Error finding container db4edf97529e9e0d6ab7a12b14a64ead2ae9b7993b1f1513e5c50a95bebbc77c: Status 404 returned error can't find the container with id db4edf97529e9e0d6ab7a12b14a64ead2ae9b7993b1f1513e5c50a95bebbc77c Jul 28 10:34:31.193961 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:31.193879 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" event={"ID":"0155405c-6cda-4869-b9da-7ba7d508058d","Type":"ContainerStarted","Data":"db4edf97529e9e0d6ab7a12b14a64ead2ae9b7993b1f1513e5c50a95bebbc77c"} Jul 28 10:34:32.035244 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:32.035207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:32.140285 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:32.140250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:33.203709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:33.203667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" event={"ID":"0155405c-6cda-4869-b9da-7ba7d508058d","Type":"ContainerStarted","Data":"f48398cd9021fa07050d602b3567fc572119765868aba88c7eff9d3939dab4d7"} Jul 28 10:34:33.225017 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:33.224965 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vbc7" podStartSLOduration=1.800107114 podStartE2EDuration="3.224950051s" totalImagesPullingTime="1.424842937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:33.223171578 +0000 UTC m=+670.355830889" watchObservedRunningTime="2026-07-28 10:34:33.224950051 +0000 UTC m=+670.357609330" Jul 28 10:34:33.605012 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:33.604971 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:33.733345 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:33.733316 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:33.944278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:33.944213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:34.044271 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.044235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:34.058627 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.058599 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqb4c"] Jul 28 10:34:34.060685 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.060669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.065080 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.065057 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 10:34:34.065198 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.065137 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 10:34:34.076449 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.076429 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqb4c"] Jul 28 10:34:34.126023 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.125997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.126128 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.126027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27gp\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-kube-api-access-v27gp\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.227237 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.227213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.227537 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.227244 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27gp\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-kube-api-access-v27gp\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.242286 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.242256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.242528 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.242510 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27gp\" (UniqueName: \"kubernetes.io/projected/76a3a96e-0174-4576-ad4a-62ffdc1704d4-kube-api-access-v27gp\") pod \"cert-manager-webhook-6486ffd997-cqb4c\" (UID: \"76a3a96e-0174-4576-ad4a-62ffdc1704d4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.383774 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.383752 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:34.503789 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.503705 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr"] Jul 28 10:34:34.507129 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.507106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.508048 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.508032 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cqb4c"] Jul 28 10:34:34.510072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.510050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 10:34:34.510162 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.510050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 10:34:34.510162 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.510107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knqhc\"" Jul 28 10:34:34.510654 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:34.510629 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76a3a96e_0174_4576_ad4a_62ffdc1704d4.slice/crio-c535c74ac483607cd979b244aeb26c3eff35c71f7b776e8cd0ab7d761b8019b3 WatchSource:0}: Error finding container c535c74ac483607cd979b244aeb26c3eff35c71f7b776e8cd0ab7d761b8019b3: Status 404 returned error can't find the container with id c535c74ac483607cd979b244aeb26c3eff35c71f7b776e8cd0ab7d761b8019b3 Jul 28 10:34:34.517008 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.516981 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr"] Jul 28 10:34:34.631601 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.631576 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqstf\" (UniqueName: \"kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.631713 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.631615 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.631713 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.631636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.732519 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.732491 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqstf\" (UniqueName: \"kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.732644 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.732538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.732644 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.732558 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.732879 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.732861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.732932 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.732897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.740783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.740766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqstf\" (UniqueName: \"kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.823772 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.823709 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:34.948886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:34.948858 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr"] Jul 28 10:34:34.950891 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:34.950865 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4548c9_8898_4574_b61e_e79c939de929.slice/crio-cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047 WatchSource:0}: Error finding container cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047: Status 404 returned error can't find the container with id cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047 Jul 28 10:34:35.214956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:35.214924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" event={"ID":"76a3a96e-0174-4576-ad4a-62ffdc1704d4","Type":"ContainerStarted","Data":"c535c74ac483607cd979b244aeb26c3eff35c71f7b776e8cd0ab7d761b8019b3"} Jul 28 10:34:35.216654 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:35.216625 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c4548c9-8898-4574-b61e-e79c939de929" containerID="26276f0c5af3f860805c8d6536d928e8fdb5af252190bb64aaf7948063a4c330" exitCode=0 Jul 28 10:34:35.216811 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:35.216663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" event={"ID":"7c4548c9-8898-4574-b61e-e79c939de929","Type":"ContainerDied","Data":"26276f0c5af3f860805c8d6536d928e8fdb5af252190bb64aaf7948063a4c330"} Jul 28 10:34:35.216811 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:35.216696 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" event={"ID":"7c4548c9-8898-4574-b61e-e79c939de929","Type":"ContainerStarted","Data":"cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047"} Jul 28 10:34:36.149458 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:36.149406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:36.338847 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:36.338807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:38.233634 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:38.233604 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c4548c9-8898-4574-b61e-e79c939de929" containerID="e7e7edbe20db4844274fa0b4f4c3f772f85b8f1b725cdbd3b07043dfe882a072" exitCode=0 Jul 28 10:34:38.234068 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:38.233699 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" event={"ID":"7c4548c9-8898-4574-b61e-e79c939de929","Type":"ContainerDied","Data":"e7e7edbe20db4844274fa0b4f4c3f772f85b8f1b725cdbd3b07043dfe882a072"} Jul 28 10:34:38.235300 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:38.235277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" event={"ID":"76a3a96e-0174-4576-ad4a-62ffdc1704d4","Type":"ContainerStarted","Data":"6c205a2ccff418f7eedc7dba2cc90196597f7120c434bcf841bf668806574d1a"} Jul 28 10:34:38.235479 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:38.235462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:38.267752 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:38.267699 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" podStartSLOduration=0.940128709 podStartE2EDuration="4.267684322s" totalImagesPullingTime="3.327555613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:38.26636206 +0000 UTC m=+675.399021352" watchObservedRunningTime="2026-07-28 10:34:38.267684322 +0000 UTC m=+675.400343601" Jul 28 10:34:39.023769 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:39.023724 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:39.078924 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:39.078895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:39.241884 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:39.241852 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c4548c9-8898-4574-b61e-e79c939de929" containerID="04bdc3f7206374bed9c417f365124d741a386a25f28593c8265fb17951854319" exitCode=0 Jul 28 10:34:39.242253 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:39.241934 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" event={"ID":"7c4548c9-8898-4574-b61e-e79c939de929","Type":"ContainerDied","Data":"04bdc3f7206374bed9c417f365124d741a386a25f28593c8265fb17951854319"} Jul 28 10:34:40.370798 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.370777 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:40.476956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.476933 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle\") pod \"7c4548c9-8898-4574-b61e-e79c939de929\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " Jul 28 10:34:40.477100 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.477004 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqstf\" (UniqueName: \"kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf\") pod \"7c4548c9-8898-4574-b61e-e79c939de929\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " Jul 28 10:34:40.477100 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.477032 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util\") pod \"7c4548c9-8898-4574-b61e-e79c939de929\" (UID: \"7c4548c9-8898-4574-b61e-e79c939de929\") " Jul 28 10:34:40.477397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.477372 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle" (OuterVolumeSpecName: "bundle") pod "7c4548c9-8898-4574-b61e-e79c939de929" (UID: "7c4548c9-8898-4574-b61e-e79c939de929"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:34:40.479201 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.479180 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf" (OuterVolumeSpecName: "kube-api-access-gqstf") pod "7c4548c9-8898-4574-b61e-e79c939de929" (UID: "7c4548c9-8898-4574-b61e-e79c939de929"). InnerVolumeSpecName "kube-api-access-gqstf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:34:40.481793 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.481768 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util" (OuterVolumeSpecName: "util") pod "7c4548c9-8898-4574-b61e-e79c939de929" (UID: "7c4548c9-8898-4574-b61e-e79c939de929"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:34:40.577717 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.577665 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqstf\" (UniqueName: \"kubernetes.io/projected/7c4548c9-8898-4574-b61e-e79c939de929-kube-api-access-gqstf\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:40.577717 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.577686 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:40.577717 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:40.577698 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4548c9-8898-4574-b61e-e79c939de929-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:34:41.253314 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:41.253279 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" event={"ID":"7c4548c9-8898-4574-b61e-e79c939de929","Type":"ContainerDied","Data":"cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047"} Jul 28 10:34:41.253482 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:41.253336 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf377a807060b3df8949a0e26d50bd3fbc4f8a585a6c1fdbaef50461d8c2b047" Jul 28 10:34:41.253482 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:41.253352 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv9glr" Jul 28 10:34:41.732380 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:41.732342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:41.915638 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:41.915609 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:44.244025 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.243996 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cqb4c" Jul 28 10:34:44.351102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.351062 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:44.542840 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.542759 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-td9cj"] Jul 28 10:34:44.543134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543120 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="pull" Jul 28 10:34:44.543179 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543137 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="pull" Jul 28 10:34:44.543179 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543145 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="extract" Jul 28 10:34:44.543179 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543152 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="extract" Jul 28 10:34:44.543179 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543159 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="util" Jul 28 10:34:44.543179 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543165 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="util" Jul 28 10:34:44.543324 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.543224 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4548c9-8898-4574-b61e-e79c939de929" containerName="extract" Jul 28 10:34:44.547032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.547018 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.550862 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.550841 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wvxw6\"" Jul 28 10:34:44.562841 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.562817 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-td9cj"] Jul 28 10:34:44.707089 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.707057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-bound-sa-token\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.707216 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.707103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8p6\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-kube-api-access-vp8p6\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.808388 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.808329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-bound-sa-token\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.808388 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.808358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8p6\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-kube-api-access-vp8p6\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.816390 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.816360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8p6\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-kube-api-access-vp8p6\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.816390 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.816373 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b506a26c-6f5a-4da8-a81c-b23130cc453f-bound-sa-token\") pod \"cert-manager-55b4d954d5-td9cj\" (UID: \"b506a26c-6f5a-4da8-a81c-b23130cc453f\") " pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.855419 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.855402 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-td9cj" Jul 28 10:34:44.982236 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:44.982211 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-td9cj"] Jul 28 10:34:44.984938 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:44.984906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb506a26c_6f5a_4da8_a81c_b23130cc453f.slice/crio-19abb98072159aa5d810d1d74588e94c4c2820738feed7021b45537e55d89039 WatchSource:0}: Error finding container 19abb98072159aa5d810d1d74588e94c4c2820738feed7021b45537e55d89039: Status 404 returned error can't find the container with id 19abb98072159aa5d810d1d74588e94c4c2820738feed7021b45537e55d89039 Jul 28 10:34:45.271783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:45.271724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-td9cj" event={"ID":"b506a26c-6f5a-4da8-a81c-b23130cc453f","Type":"ContainerStarted","Data":"f1278cbe59cb1c982aeb5bdad88f8e44fc14c158989e84a35d0f62d5b475c157"} Jul 28 10:34:45.271783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:45.271788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-td9cj" event={"ID":"b506a26c-6f5a-4da8-a81c-b23130cc453f","Type":"ContainerStarted","Data":"19abb98072159aa5d810d1d74588e94c4c2820738feed7021b45537e55d89039"} Jul 28 10:34:45.288029 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:45.287985 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-td9cj" podStartSLOduration=1.287971991 podStartE2EDuration="1.287971991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:45.286695416 +0000 UTC m=+682.419354695" watchObservedRunningTime="2026-07-28 10:34:45.287971991 +0000 UTC m=+682.420631271" Jul 28 10:34:45.428000 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:45.427970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:45.549481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:45.549408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:47.445489 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.445458 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq"] Jul 28 10:34:47.448664 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.448643 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.451134 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.451113 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:34:47.452350 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.452329 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 10:34:47.452509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.452411 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-brbch\"" Jul 28 10:34:47.462847 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.460202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq"] Jul 28 10:34:47.527085 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.527063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phppx\" (UniqueName: \"kubernetes.io/projected/55072a23-0aea-480c-a450-411bc7272c3a-kube-api-access-phppx\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.527206 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.527189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55072a23-0aea-480c-a450-411bc7272c3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.627769 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.627744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55072a23-0aea-480c-a450-411bc7272c3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.627843 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.627782 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phppx\" (UniqueName: \"kubernetes.io/projected/55072a23-0aea-480c-a450-411bc7272c3a-kube-api-access-phppx\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.628072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.628054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55072a23-0aea-480c-a450-411bc7272c3a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.637199 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.637175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phppx\" (UniqueName: \"kubernetes.io/projected/55072a23-0aea-480c-a450-411bc7272c3a-kube-api-access-phppx\") pod \"openshift-lws-operator-fd8ccff4c-ddnrq\" (UID: \"55072a23-0aea-480c-a450-411bc7272c3a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.759561 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.759539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" Jul 28 10:34:47.886426 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:47.886399 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq"] Jul 28 10:34:47.889074 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:47.889048 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55072a23_0aea_480c_a450_411bc7272c3a.slice/crio-b65051e4d88f66e17936b7def7852c79c20c38d0e6bd1bb24df6f79eae8c9265 WatchSource:0}: Error finding container b65051e4d88f66e17936b7def7852c79c20c38d0e6bd1bb24df6f79eae8c9265: Status 404 returned error can't find the container with id b65051e4d88f66e17936b7def7852c79c20c38d0e6bd1bb24df6f79eae8c9265 Jul 28 10:34:48.285847 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:48.285815 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" event={"ID":"55072a23-0aea-480c-a450-411bc7272c3a","Type":"ContainerStarted","Data":"b65051e4d88f66e17936b7def7852c79c20c38d0e6bd1bb24df6f79eae8c9265"} Jul 28 10:34:50.298099 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:50.298061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" event={"ID":"55072a23-0aea-480c-a450-411bc7272c3a","Type":"ContainerStarted","Data":"5386c134fe70d1fcb8f258f885d933991859f3730065bcd2b023bd1314bfb7ca"} Jul 28 10:34:50.315384 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:50.315336 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddnrq" podStartSLOduration=1.618574078 podStartE2EDuration="3.315321189s" totalImagesPullingTime="1.696747111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:50.314829426 +0000 UTC m=+687.447488705" watchObservedRunningTime="2026-07-28 10:34:50.315321189 +0000 UTC m=+687.447980479" Jul 28 10:34:51.153799 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:51.153760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:51.259716 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:51.259676 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:53.083349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:53.083315 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:34:56.302111 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.302075 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb"] Jul 28 10:34:56.305597 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.305580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.308333 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.308311 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 10:34:56.308445 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.308316 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-knqhc\"" Jul 28 10:34:56.309463 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.309448 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 10:34:56.313557 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.313532 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb"] Jul 28 10:34:56.489975 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.489951 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmn46\" (UniqueName: \"kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.490107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.489999 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.490107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.490019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.590838 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.590767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmn46\" (UniqueName: \"kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.590838 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.590828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.590955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.590847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.591151 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.591136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.591226 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.591206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.599784 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.599757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmn46\" (UniqueName: \"kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.615348 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.615327 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:34:56.740647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:56.740622 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb"] Jul 28 10:34:56.742527 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:34:56.742503 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f61faf_81a9_4597_b95b_3c956d5001bb.slice/crio-930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39 WatchSource:0}: Error finding container 930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39: Status 404 returned error can't find the container with id 930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39 Jul 28 10:34:57.329851 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:57.329820 2567 generic.go:356] "Generic (PLEG): container finished" podID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerID="3e90f7b04846401425f60216a2a282ec53f741cfccdb02e5d1bb6ecbc23ddf58" exitCode=0 Jul 28 10:34:57.330187 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:57.329901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" event={"ID":"38f61faf-81a9-4597-b95b-3c956d5001bb","Type":"ContainerDied","Data":"3e90f7b04846401425f60216a2a282ec53f741cfccdb02e5d1bb6ecbc23ddf58"} Jul 28 10:34:57.330187 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:57.329941 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" event={"ID":"38f61faf-81a9-4597-b95b-3c956d5001bb","Type":"ContainerStarted","Data":"930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39"} Jul 28 10:34:58.336332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:58.336301 2567 generic.go:356] "Generic (PLEG): container finished" podID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerID="fcdb400ac9b1cc0d82e963aff88af84f136af2c5b1dc6aa2edc5b6d02975d990" exitCode=0 Jul 28 10:34:58.336680 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:58.336375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" event={"ID":"38f61faf-81a9-4597-b95b-3c956d5001bb","Type":"ContainerDied","Data":"fcdb400ac9b1cc0d82e963aff88af84f136af2c5b1dc6aa2edc5b6d02975d990"} Jul 28 10:34:59.343220 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:59.343188 2567 generic.go:356] "Generic (PLEG): container finished" podID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerID="31cf6398daf9b6b77d20f934b67bf53d35f778d38a42b787670f80bb911ddcc3" exitCode=0 Jul 28 10:34:59.343657 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:34:59.343275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" event={"ID":"38f61faf-81a9-4597-b95b-3c956d5001bb","Type":"ContainerDied","Data":"31cf6398daf9b6b77d20f934b67bf53d35f778d38a42b787670f80bb911ddcc3"} Jul 28 10:35:00.475615 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.475591 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:35:00.525188 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.525156 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmn46\" (UniqueName: \"kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46\") pod \"38f61faf-81a9-4597-b95b-3c956d5001bb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " Jul 28 10:35:00.525297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.525239 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util\") pod \"38f61faf-81a9-4597-b95b-3c956d5001bb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " Jul 28 10:35:00.525297 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.525293 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle\") pod \"38f61faf-81a9-4597-b95b-3c956d5001bb\" (UID: \"38f61faf-81a9-4597-b95b-3c956d5001bb\") " Jul 28 10:35:00.525977 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.525948 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle" (OuterVolumeSpecName: "bundle") pod "38f61faf-81a9-4597-b95b-3c956d5001bb" (UID: "38f61faf-81a9-4597-b95b-3c956d5001bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:35:00.527246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.527229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46" (OuterVolumeSpecName: "kube-api-access-tmn46") pod "38f61faf-81a9-4597-b95b-3c956d5001bb" (UID: "38f61faf-81a9-4597-b95b-3c956d5001bb"). InnerVolumeSpecName "kube-api-access-tmn46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:35:00.533552 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.533474 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util" (OuterVolumeSpecName: "util") pod "38f61faf-81a9-4597-b95b-3c956d5001bb" (UID: "38f61faf-81a9-4597-b95b-3c956d5001bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:35:00.626358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.626304 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:35:00.626358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.626326 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38f61faf-81a9-4597-b95b-3c956d5001bb-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:35:00.626358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:00.626334 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmn46\" (UniqueName: \"kubernetes.io/projected/38f61faf-81a9-4597-b95b-3c956d5001bb-kube-api-access-tmn46\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:35:01.354375 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:01.354335 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" event={"ID":"38f61faf-81a9-4597-b95b-3c956d5001bb","Type":"ContainerDied","Data":"930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39"} Jul 28 10:35:01.354375 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:01.354374 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9k7zb" Jul 28 10:35:01.354578 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:01.354424 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="930c1a7aadb51ab5f2d5a77f11f1a5988dd7528181b34e8cba6f7c2abbcb0a39" Jul 28 10:35:06.641974 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:06.641942 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:06.735349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:06.735321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.252268 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.252239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.525132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.525047 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.614539 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.614506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.646391 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646367 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k"] Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646657 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="util" Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646676 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="util" Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646687 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="extract" Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646693 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="extract" Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646708 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="pull" Jul 28 10:35:07.646710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646713 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="pull" Jul 28 10:35:07.646924 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.646777 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f61faf-81a9-4597-b95b-3c956d5001bb" containerName="extract" Jul 28 10:35:07.668938 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.668916 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k"] Jul 28 10:35:07.669050 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.669005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.674064 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674030 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 10:35:07.674183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674066 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k7g2n\"" Jul 28 10:35:07.674183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.674183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.674397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzdx\" (UniqueName: \"kubernetes.io/projected/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-kube-api-access-dzzdx\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.674397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674217 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 10:35:07.674397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.674397 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.674359 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.676029 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.675673 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 10:35:07.676356 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.676340 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 10:35:07.775479 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzdx\" (UniqueName: \"kubernetes.io/projected/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-kube-api-access-dzzdx\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.775479 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775462 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.775636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.775636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.775636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.775904 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.775876 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.777808 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.777780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.778032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.778014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.778107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.778068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.795192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.795172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzdx\" (UniqueName: \"kubernetes.io/projected/47faa125-f9cf-4372-8e33-e2b5d6fc2b39-kube-api-access-dzzdx\") pod \"opendatahub-operator-controller-manager-559f66f7dc-v2b8k\" (UID: \"47faa125-f9cf-4372-8e33-e2b5d6fc2b39\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:07.832121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.832089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.928824 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.928800 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:07.979917 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:07.979896 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:08.028460 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.028406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:08.127112 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.127080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:08.136822 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.136802 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k"] Jul 28 10:35:08.138839 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:35:08.138817 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47faa125_f9cf_4372_8e33_e2b5d6fc2b39.slice/crio-dfdd59f52998b07305400951cc8226063ac03e853fb7bbc03efec8aac93d6d00 WatchSource:0}: Error finding container dfdd59f52998b07305400951cc8226063ac03e853fb7bbc03efec8aac93d6d00: Status 404 returned error can't find the container with id dfdd59f52998b07305400951cc8226063ac03e853fb7bbc03efec8aac93d6d00 Jul 28 10:35:08.193828 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.193806 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:08.292923 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.292851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:08.389142 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.389117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" event={"ID":"47faa125-f9cf-4372-8e33-e2b5d6fc2b39","Type":"ContainerStarted","Data":"dfdd59f52998b07305400951cc8226063ac03e853fb7bbc03efec8aac93d6d00"} Jul 28 10:35:08.392784 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:08.392756 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:11.406826 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:11.406724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" event={"ID":"47faa125-f9cf-4372-8e33-e2b5d6fc2b39","Type":"ContainerStarted","Data":"f7e756dae39fafbc89fed85c0f63c980bc2ac4c07a7ffad2a2075d4342028b3a"} Jul 28 10:35:12.412572 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:12.412540 2567 generic.go:356] "Generic (PLEG): container finished" podID="47faa125-f9cf-4372-8e33-e2b5d6fc2b39" containerID="f7e756dae39fafbc89fed85c0f63c980bc2ac4c07a7ffad2a2075d4342028b3a" exitCode=0 Jul 28 10:35:12.412936 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:12.412635 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" event={"ID":"47faa125-f9cf-4372-8e33-e2b5d6fc2b39","Type":"ContainerDied","Data":"f7e756dae39fafbc89fed85c0f63c980bc2ac4c07a7ffad2a2075d4342028b3a"} Jul 28 10:35:13.419551 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:13.419507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" event={"ID":"47faa125-f9cf-4372-8e33-e2b5d6fc2b39","Type":"ContainerStarted","Data":"fef67808a4491515eb5bc6f828fb2e5a64508ef007f5609f346469d515e85286"} Jul 28 10:35:13.420049 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:13.419608 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:13.444387 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:13.444341 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" podStartSLOduration=2.114988619 podStartE2EDuration="6.444326117s" totalImagesPullingTime="3.26320834s" totalInitContainerRuntime="1.066129158s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:13.442266728 +0000 UTC m=+710.574926007" watchObservedRunningTime="2026-07-28 10:35:13.444326117 +0000 UTC m=+710.576985396" Jul 28 10:35:21.356728 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:21.356634 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:21.458408 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:21.458367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:24.426327 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:24.426296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-v2b8k" Jul 28 10:35:25.781645 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.781611 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc"] Jul 28 10:35:25.787983 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.787965 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.793083 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.793054 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 10:35:25.793083 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.793077 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 10:35:25.793237 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.793059 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 10:35:25.793237 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.793107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zr6t4\"" Jul 28 10:35:25.798056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.798034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc"] Jul 28 10:35:25.815720 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.815698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.815832 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.815750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bda75dab-ae79-4485-a8b0-ee4385b6630d-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.815832 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.815784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgshw\" (UniqueName: \"kubernetes.io/projected/bda75dab-ae79-4485-a8b0-ee4385b6630d-kube-api-access-wgshw\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.815832 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.815802 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.916278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.916253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgshw\" (UniqueName: \"kubernetes.io/projected/bda75dab-ae79-4485-a8b0-ee4385b6630d-kube-api-access-wgshw\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.916400 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.916287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.916400 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.916328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.916400 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.916367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bda75dab-ae79-4485-a8b0-ee4385b6630d-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.917003 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.916982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bda75dab-ae79-4485-a8b0-ee4385b6630d-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.919049 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.919027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.919133 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.919095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bda75dab-ae79-4485-a8b0-ee4385b6630d-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:25.937675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:25.935044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgshw\" (UniqueName: \"kubernetes.io/projected/bda75dab-ae79-4485-a8b0-ee4385b6630d-kube-api-access-wgshw\") pod \"lws-controller-manager-7d8dfbcf79-tvlhc\" (UID: \"bda75dab-ae79-4485-a8b0-ee4385b6630d\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:26.023515 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.023482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:26.097387 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.097326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:26.193647 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.193613 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:26.256437 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.256407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc"] Jul 28 10:35:26.259005 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:35:26.258974 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda75dab_ae79_4485_a8b0_ee4385b6630d.slice/crio-aa17d741bb79aa7f47e57706c74aa92fc62d7f395da0ab28505f70c58f496261 WatchSource:0}: Error finding container aa17d741bb79aa7f47e57706c74aa92fc62d7f395da0ab28505f70c58f496261: Status 404 returned error can't find the container with id aa17d741bb79aa7f47e57706c74aa92fc62d7f395da0ab28505f70c58f496261 Jul 28 10:35:26.478128 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.478094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" event={"ID":"bda75dab-ae79-4485-a8b0-ee4385b6630d","Type":"ContainerStarted","Data":"aa17d741bb79aa7f47e57706c74aa92fc62d7f395da0ab28505f70c58f496261"} Jul 28 10:35:26.803974 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.803867 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:26.898389 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.898351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:26.993301 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:26.993269 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:27.385468 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:27.385410 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:27.751349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:27.751308 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:28.328917 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:28.328879 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:28.490372 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:28.490340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" event={"ID":"bda75dab-ae79-4485-a8b0-ee4385b6630d","Type":"ContainerStarted","Data":"d7f41f0a3dffaf9948789e4b4da65b7372c611a92f9646eb19452e1e2395686d"} Jul 28 10:35:28.490560 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:28.490415 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:28.507499 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:28.507455 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" podStartSLOduration=2.044680072 podStartE2EDuration="3.507442366s" totalImagesPullingTime="1.462762294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:28.506774437 +0000 UTC m=+725.639433720" watchObservedRunningTime="2026-07-28 10:35:28.507442366 +0000 UTC m=+725.640101644" Jul 28 10:35:36.481595 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:36.481559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:36.606531 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:36.606495 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:36.743225 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:36.743143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:36.870285 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:36.870248 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:37.096514 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:37.096412 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:37.481047 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:37.481020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:37.827475 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:37.827390 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:38.015035 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:38.015003 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:38.133407 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:38.133328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:39.496777 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.496719 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-tvlhc" Jul 28 10:35:39.552692 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.552643 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:39.556272 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.556240 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb"] Jul 28 10:35:39.563121 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.563097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.566890 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.566871 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 10:35:39.566999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.566971 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 10:35:39.567218 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.567201 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 10:35:39.567284 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.567265 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 10:35:39.567649 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.567627 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 10:35:39.567722 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.567684 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 10:35:39.568240 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.568223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p8bx8\"" Jul 28 10:35:39.597730 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.597705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb"] Jul 28 10:35:39.641787 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.641761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.641897 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.641793 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824fk\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-kube-api-access-824fk\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.641897 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.641823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.641897 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.641847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.642034 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.641965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.642034 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.642017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.642133 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.642103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742695 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742668 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742695 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824fk\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-kube-api-access-824fk\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.742915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.742875 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.743475 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.743413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.745223 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.745203 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.745321 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.745228 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.745541 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.745524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.745578 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.745528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.759190 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.759131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824fk\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-kube-api-access-824fk\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.761419 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.761393 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e63ffc-3903-4e58-ba63-0ceac4f58f4c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w2lzb\" (UID: \"11e63ffc-3903-4e58-ba63-0ceac4f58f4c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:39.762907 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.762881 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:39.873436 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:39.873404 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:40.062231 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:40.061590 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb"] Jul 28 10:35:40.068876 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:35:40.068807 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e63ffc_3903_4e58_ba63_0ceac4f58f4c.slice/crio-75f82af6c83552850b47189f8c8378c654c8900294eca13ff07a0ee35f727869 WatchSource:0}: Error finding container 75f82af6c83552850b47189f8c8378c654c8900294eca13ff07a0ee35f727869: Status 404 returned error can't find the container with id 75f82af6c83552850b47189f8c8378c654c8900294eca13ff07a0ee35f727869 Jul 28 10:35:40.546537 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:40.546497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" event={"ID":"11e63ffc-3903-4e58-ba63-0ceac4f58f4c","Type":"ContainerStarted","Data":"75f82af6c83552850b47189f8c8378c654c8900294eca13ff07a0ee35f727869"} Jul 28 10:35:40.769884 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:40.769848 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:42.582178 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:42.582142 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 28 10:35:42.582550 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:42.582222 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 28 10:35:43.190117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.190048 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:43.290585 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.290553 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:43.563673 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.563641 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" event={"ID":"11e63ffc-3903-4e58-ba63-0ceac4f58f4c","Type":"ContainerStarted","Data":"d64aae629e7b12fdb494b3067da38f7e5a520a88016840e35875a7ea58a99746"} Jul 28 10:35:43.563871 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.563824 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:43.565537 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.565511 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-w2lzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 10:35:43.565667 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.565559 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" podUID="11e63ffc-3903-4e58-ba63-0ceac4f58f4c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:35:43.587013 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.586966 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" podStartSLOduration=2.076374941 podStartE2EDuration="4.586951951s" totalImagesPullingTime="2.51057701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:43.586498894 +0000 UTC m=+740.719158173" watchObservedRunningTime="2026-07-28 10:35:43.586951951 +0000 UTC m=+740.719611229" Jul 28 10:35:43.818237 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.818148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:43.986506 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:43.986477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:44.084458 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:44.084374 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:35:44.569656 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:44.569629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w2lzb" Jul 28 10:35:57.142940 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:35:57.142905 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:27.514434 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:27.514396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:27.579254 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:27.579215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:27.671315 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:27.671285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:28.422014 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:28.421979 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:28.623984 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:28.623939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:36:43.519163 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.519133 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx"] Jul 28 10:36:43.523156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.523138 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.526046 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.526027 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f66n2\"" Jul 28 10:36:43.526173 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.526050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 10:36:43.526173 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.526058 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 10:36:43.531660 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.531634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx"] Jul 28 10:36:43.632911 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.632881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.633035 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.632989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.633035 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.633018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kxh\" (UniqueName: \"kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.734296 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.734250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.734449 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.734371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.734449 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.734401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kxh\" (UniqueName: \"kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.734624 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.734602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.734699 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.734673 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.743609 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.743580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kxh\" (UniqueName: \"kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.833789 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.833697 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:43.963313 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:43.963287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx"] Jul 28 10:36:43.965247 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:36:43.965195 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936f30cc_3c31_4250_9f79_3996a708ed83.slice/crio-e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe WatchSource:0}: Error finding container e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe: Status 404 returned error can't find the container with id e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe Jul 28 10:36:44.244154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.244129 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr"] Jul 28 10:36:44.247464 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.247448 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.255490 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.255464 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr"] Jul 28 10:36:44.339142 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.339117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.339258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.339158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.339258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.339178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgnp\" (UniqueName: \"kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.440395 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.440366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.440496 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.440401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgnp\" (UniqueName: \"kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.440571 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.440543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.440762 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.440722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.440916 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.440899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.449186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.449169 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgnp\" (UniqueName: \"kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.556828 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.556777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:44.684862 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.684834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr"] Jul 28 10:36:44.686239 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:36:44.686208 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ed5711_83b2_4d9c_96e9_f637d213f034.slice/crio-a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e WatchSource:0}: Error finding container a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e: Status 404 returned error can't find the container with id a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e Jul 28 10:36:44.836957 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.836899 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr"] Jul 28 10:36:44.840125 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.840108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:44.848840 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.848811 2567 generic.go:356] "Generic (PLEG): container finished" podID="936f30cc-3c31-4250-9f79-3996a708ed83" containerID="217eca3cd2198e02bfa6187daa1220827c8bf9b6fbe8493d8c03e66a60886dc2" exitCode=0 Jul 28 10:36:44.848956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.848892 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" event={"ID":"936f30cc-3c31-4250-9f79-3996a708ed83","Type":"ContainerDied","Data":"217eca3cd2198e02bfa6187daa1220827c8bf9b6fbe8493d8c03e66a60886dc2"} Jul 28 10:36:44.848956 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.848939 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" event={"ID":"936f30cc-3c31-4250-9f79-3996a708ed83","Type":"ContainerStarted","Data":"e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe"} Jul 28 10:36:44.849191 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.849144 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr"] Jul 28 10:36:44.851129 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.851069 2567 generic.go:356] "Generic (PLEG): container finished" podID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerID="fda587fc99a94d99cf1bdffeb3009fe409d82c09bac7d8d66572b6f9624fb2b9" exitCode=0 Jul 28 10:36:44.851256 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.851142 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" event={"ID":"d3ed5711-83b2-4d9c-96e9-f637d213f034","Type":"ContainerDied","Data":"fda587fc99a94d99cf1bdffeb3009fe409d82c09bac7d8d66572b6f9624fb2b9"} Jul 28 10:36:44.851256 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.851187 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" event={"ID":"d3ed5711-83b2-4d9c-96e9-f637d213f034","Type":"ContainerStarted","Data":"a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e"} Jul 28 10:36:44.944643 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.944619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:44.944805 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.944657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84297\" (UniqueName: \"kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:44.944881 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:44.944855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.046074 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.046047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.046183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.046136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.046258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.046180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84297\" (UniqueName: \"kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.046459 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.046436 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.046534 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.046477 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.054861 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.054842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84297\" (UniqueName: \"kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.156483 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.156428 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:45.238262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.238230 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk"] Jul 28 10:36:45.242960 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.242849 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.250278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.250160 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk"] Jul 28 10:36:45.298914 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.298892 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr"] Jul 28 10:36:45.350113 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.350085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.350228 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.350173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xdz\" (UniqueName: \"kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.350228 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.350219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.451198 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.451125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xdz\" (UniqueName: \"kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.451198 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.451183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.451362 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.451254 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.451706 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.451683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.451841 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.451759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.460473 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.460450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xdz\" (UniqueName: \"kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.558391 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.558360 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:45.810221 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.810193 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk"] Jul 28 10:36:45.813584 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:36:45.813560 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ae250_366c_4b93_ba4d_843c38ac2eb7.slice/crio-7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5 WatchSource:0}: Error finding container 7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5: Status 404 returned error can't find the container with id 7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5 Jul 28 10:36:45.857460 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.857438 2567 generic.go:356] "Generic (PLEG): container finished" podID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerID="055acbf85aa6de75ba2ceb739cbec34219c6b87deb7a223a33ca89beebf34f85" exitCode=0 Jul 28 10:36:45.857550 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.857510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" event={"ID":"d3ed5711-83b2-4d9c-96e9-f637d213f034","Type":"ContainerDied","Data":"055acbf85aa6de75ba2ceb739cbec34219c6b87deb7a223a33ca89beebf34f85"} Jul 28 10:36:45.858944 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.858908 2567 generic.go:356] "Generic (PLEG): container finished" podID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerID="74cf31be39fec5e392a21e0460ee5b1bf16331233f626cf1be8ffb30ac08f06e" exitCode=0 Jul 28 10:36:45.859007 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.858970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" event={"ID":"40f011ac-ad41-4f85-ab22-b760f188eb9b","Type":"ContainerDied","Data":"74cf31be39fec5e392a21e0460ee5b1bf16331233f626cf1be8ffb30ac08f06e"} Jul 28 10:36:45.859059 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.859018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" event={"ID":"40f011ac-ad41-4f85-ab22-b760f188eb9b","Type":"ContainerStarted","Data":"4fa5841f485469a83b99b1314bc906b487f7ce57af3b68605f70d22bb620d3e5"} Jul 28 10:36:45.860959 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.860935 2567 generic.go:356] "Generic (PLEG): container finished" podID="936f30cc-3c31-4250-9f79-3996a708ed83" containerID="d75b7160cb1efcdce29c18f65746de6f5be7401c460e26397d2defc89ee0bdc2" exitCode=0 Jul 28 10:36:45.861056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.861011 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" event={"ID":"936f30cc-3c31-4250-9f79-3996a708ed83","Type":"ContainerDied","Data":"d75b7160cb1efcdce29c18f65746de6f5be7401c460e26397d2defc89ee0bdc2"} Jul 28 10:36:45.862422 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:45.862304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" event={"ID":"b83ae250-366c-4b93-ba4d-843c38ac2eb7","Type":"ContainerStarted","Data":"7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5"} Jul 28 10:36:46.869440 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.869405 2567 generic.go:356] "Generic (PLEG): container finished" podID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerID="5ed0be0694e17beeb672a925aaafb4eff66c269799c0acd00d77980e11eb1394" exitCode=0 Jul 28 10:36:46.869799 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.869480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" event={"ID":"d3ed5711-83b2-4d9c-96e9-f637d213f034","Type":"ContainerDied","Data":"5ed0be0694e17beeb672a925aaafb4eff66c269799c0acd00d77980e11eb1394"} Jul 28 10:36:46.871181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.871159 2567 generic.go:356] "Generic (PLEG): container finished" podID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerID="3b191844fd6c96028c7804ca4444be5e60a7df1824e8016d1bf5eadd7517c79a" exitCode=0 Jul 28 10:36:46.871277 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.871238 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" event={"ID":"40f011ac-ad41-4f85-ab22-b760f188eb9b","Type":"ContainerDied","Data":"3b191844fd6c96028c7804ca4444be5e60a7df1824e8016d1bf5eadd7517c79a"} Jul 28 10:36:46.873326 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.873305 2567 generic.go:356] "Generic (PLEG): container finished" podID="936f30cc-3c31-4250-9f79-3996a708ed83" containerID="017312c98cecc8483cdb02c97ed21d79ef5fb078aefd5591719c3db74d937967" exitCode=0 Jul 28 10:36:46.873423 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.873384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" event={"ID":"936f30cc-3c31-4250-9f79-3996a708ed83","Type":"ContainerDied","Data":"017312c98cecc8483cdb02c97ed21d79ef5fb078aefd5591719c3db74d937967"} Jul 28 10:36:46.874876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.874855 2567 generic.go:356] "Generic (PLEG): container finished" podID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerID="025b4109aa3fc87abec795d18d5cdb305b8e65852000613c86e13696e9c4649d" exitCode=0 Jul 28 10:36:46.874954 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:46.874904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" event={"ID":"b83ae250-366c-4b93-ba4d-843c38ac2eb7","Type":"ContainerDied","Data":"025b4109aa3fc87abec795d18d5cdb305b8e65852000613c86e13696e9c4649d"} Jul 28 10:36:47.881382 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:47.881345 2567 generic.go:356] "Generic (PLEG): container finished" podID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerID="89fbd724609f57ff8ab6419a50f5ec49239e84fa60c16f3d72f767fc27462e21" exitCode=0 Jul 28 10:36:47.881728 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:47.881428 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" event={"ID":"40f011ac-ad41-4f85-ab22-b760f188eb9b","Type":"ContainerDied","Data":"89fbd724609f57ff8ab6419a50f5ec49239e84fa60c16f3d72f767fc27462e21"} Jul 28 10:36:48.064077 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.064054 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:48.067068 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.067049 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:48.177911 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.177839 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2kxh\" (UniqueName: \"kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh\") pod \"936f30cc-3c31-4250-9f79-3996a708ed83\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " Jul 28 10:36:48.178034 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.177945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle\") pod \"d3ed5711-83b2-4d9c-96e9-f637d213f034\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " Jul 28 10:36:48.178034 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178009 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cgnp\" (UniqueName: \"kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp\") pod \"d3ed5711-83b2-4d9c-96e9-f637d213f034\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " Jul 28 10:36:48.178120 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178041 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util\") pod \"936f30cc-3c31-4250-9f79-3996a708ed83\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " Jul 28 10:36:48.178171 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178153 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle\") pod \"936f30cc-3c31-4250-9f79-3996a708ed83\" (UID: \"936f30cc-3c31-4250-9f79-3996a708ed83\") " Jul 28 10:36:48.178219 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178204 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util\") pod \"d3ed5711-83b2-4d9c-96e9-f637d213f034\" (UID: \"d3ed5711-83b2-4d9c-96e9-f637d213f034\") " Jul 28 10:36:48.178507 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178477 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle" (OuterVolumeSpecName: "bundle") pod "d3ed5711-83b2-4d9c-96e9-f637d213f034" (UID: "d3ed5711-83b2-4d9c-96e9-f637d213f034"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:48.178651 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.178630 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle" (OuterVolumeSpecName: "bundle") pod "936f30cc-3c31-4250-9f79-3996a708ed83" (UID: "936f30cc-3c31-4250-9f79-3996a708ed83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:48.180343 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.180314 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh" (OuterVolumeSpecName: "kube-api-access-g2kxh") pod "936f30cc-3c31-4250-9f79-3996a708ed83" (UID: "936f30cc-3c31-4250-9f79-3996a708ed83"). InnerVolumeSpecName "kube-api-access-g2kxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:36:48.180709 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.180689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp" (OuterVolumeSpecName: "kube-api-access-2cgnp") pod "d3ed5711-83b2-4d9c-96e9-f637d213f034" (UID: "d3ed5711-83b2-4d9c-96e9-f637d213f034"). InnerVolumeSpecName "kube-api-access-2cgnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:36:48.183168 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.183146 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util" (OuterVolumeSpecName: "util") pod "936f30cc-3c31-4250-9f79-3996a708ed83" (UID: "936f30cc-3c31-4250-9f79-3996a708ed83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:48.184585 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.184563 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util" (OuterVolumeSpecName: "util") pod "d3ed5711-83b2-4d9c-96e9-f637d213f034" (UID: "d3ed5711-83b2-4d9c-96e9-f637d213f034"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:48.279719 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279694 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.279719 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279716 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgnp\" (UniqueName: \"kubernetes.io/projected/d3ed5711-83b2-4d9c-96e9-f637d213f034-kube-api-access-2cgnp\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.279861 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279726 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.279861 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279754 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/936f30cc-3c31-4250-9f79-3996a708ed83-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.279861 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279762 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3ed5711-83b2-4d9c-96e9-f637d213f034-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.279861 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.279769 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kxh\" (UniqueName: \"kubernetes.io/projected/936f30cc-3c31-4250-9f79-3996a708ed83-kube-api-access-g2kxh\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:48.888659 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.888622 2567 generic.go:356] "Generic (PLEG): container finished" podID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerID="573076024e0532a5891b03352934caea7b908d93a5ebc854564a0ba0f0dcf9ab" exitCode=0 Jul 28 10:36:48.889086 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.888706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" event={"ID":"b83ae250-366c-4b93-ba4d-843c38ac2eb7","Type":"ContainerDied","Data":"573076024e0532a5891b03352934caea7b908d93a5ebc854564a0ba0f0dcf9ab"} Jul 28 10:36:48.890925 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.890906 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" Jul 28 10:36:48.890925 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.890914 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr" event={"ID":"d3ed5711-83b2-4d9c-96e9-f637d213f034","Type":"ContainerDied","Data":"a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e"} Jul 28 10:36:48.891084 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.890959 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fa44ad7fde5c3179740a67c6d9014898ccf41cca487e768bf5b38271243b7e" Jul 28 10:36:48.893092 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.893064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" event={"ID":"936f30cc-3c31-4250-9f79-3996a708ed83","Type":"ContainerDied","Data":"e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe"} Jul 28 10:36:48.893193 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.893103 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64a13e577e1abacd9cd03a7084a3e8fd8ef13f6bc319a4397098dc61b0edbfe" Jul 28 10:36:48.893268 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:48.893244 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx" Jul 28 10:36:49.027554 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.027531 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:49.187136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.187069 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util\") pod \"40f011ac-ad41-4f85-ab22-b760f188eb9b\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " Jul 28 10:36:49.187242 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.187191 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84297\" (UniqueName: \"kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297\") pod \"40f011ac-ad41-4f85-ab22-b760f188eb9b\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " Jul 28 10:36:49.187242 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.187219 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle\") pod \"40f011ac-ad41-4f85-ab22-b760f188eb9b\" (UID: \"40f011ac-ad41-4f85-ab22-b760f188eb9b\") " Jul 28 10:36:49.187823 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.187794 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle" (OuterVolumeSpecName: "bundle") pod "40f011ac-ad41-4f85-ab22-b760f188eb9b" (UID: "40f011ac-ad41-4f85-ab22-b760f188eb9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:49.189230 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.189208 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297" (OuterVolumeSpecName: "kube-api-access-84297") pod "40f011ac-ad41-4f85-ab22-b760f188eb9b" (UID: "40f011ac-ad41-4f85-ab22-b760f188eb9b"). InnerVolumeSpecName "kube-api-access-84297". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:36:49.192374 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.192355 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util" (OuterVolumeSpecName: "util") pod "40f011ac-ad41-4f85-ab22-b760f188eb9b" (UID: "40f011ac-ad41-4f85-ab22-b760f188eb9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:49.287884 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.287847 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84297\" (UniqueName: \"kubernetes.io/projected/40f011ac-ad41-4f85-ab22-b760f188eb9b-kube-api-access-84297\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:49.287884 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.287879 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:49.287884 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.287889 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40f011ac-ad41-4f85-ab22-b760f188eb9b-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:49.900798 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.900760 2567 generic.go:356] "Generic (PLEG): container finished" podID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerID="8b45ba55637d2093d15c90db7e052b9148875c34884f7653d86b70efa5db71a3" exitCode=0 Jul 28 10:36:49.901274 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.900845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" event={"ID":"b83ae250-366c-4b93-ba4d-843c38ac2eb7","Type":"ContainerDied","Data":"8b45ba55637d2093d15c90db7e052b9148875c34884f7653d86b70efa5db71a3"} Jul 28 10:36:49.902827 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.902803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" event={"ID":"40f011ac-ad41-4f85-ab22-b760f188eb9b","Type":"ContainerDied","Data":"4fa5841f485469a83b99b1314bc906b487f7ce57af3b68605f70d22bb620d3e5"} Jul 28 10:36:49.902964 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.902831 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr" Jul 28 10:36:49.902964 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:49.902836 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa5841f485469a83b99b1314bc906b487f7ce57af3b68605f70d22bb620d3e5" Jul 28 10:36:51.040698 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.040674 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:51.201599 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.201529 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle\") pod \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " Jul 28 10:36:51.201599 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.201568 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util\") pod \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " Jul 28 10:36:51.201848 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.201614 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7xdz\" (UniqueName: \"kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz\") pod \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\" (UID: \"b83ae250-366c-4b93-ba4d-843c38ac2eb7\") " Jul 28 10:36:51.202139 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.202112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle" (OuterVolumeSpecName: "bundle") pod "b83ae250-366c-4b93-ba4d-843c38ac2eb7" (UID: "b83ae250-366c-4b93-ba4d-843c38ac2eb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:51.203876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.203852 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz" (OuterVolumeSpecName: "kube-api-access-m7xdz") pod "b83ae250-366c-4b93-ba4d-843c38ac2eb7" (UID: "b83ae250-366c-4b93-ba4d-843c38ac2eb7"). InnerVolumeSpecName "kube-api-access-m7xdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:36:51.285907 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.285880 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util" (OuterVolumeSpecName: "util") pod "b83ae250-366c-4b93-ba4d-843c38ac2eb7" (UID: "b83ae250-366c-4b93-ba4d-843c38ac2eb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:51.302955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.302929 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-util\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:51.302955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.302952 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xdz\" (UniqueName: \"kubernetes.io/projected/b83ae250-366c-4b93-ba4d-843c38ac2eb7-kube-api-access-m7xdz\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:51.303056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.302962 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b83ae250-366c-4b93-ba4d-843c38ac2eb7-bundle\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:36:51.914559 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.914530 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" event={"ID":"b83ae250-366c-4b93-ba4d-843c38ac2eb7","Type":"ContainerDied","Data":"7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5"} Jul 28 10:36:51.914559 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.914546 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk" Jul 28 10:36:51.914798 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:51.914572 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7facebdd291a8aac9515e6e4fbbaa7594191aac05cd2c3878ea42f1ba3c487c5" Jul 28 10:36:56.803065 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:36:56.803023 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:00.284410 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:00.284367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:01.813701 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:01.813661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:02.204796 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:02.204709 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:03.248943 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:03.248903 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:04.809935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:04.809900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:05.837288 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:05.837243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:07.210781 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:07.210720 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:08.449932 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:08.449894 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:10.685026 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:10.684993 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:11.023849 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:11.023819 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:11.106019 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:11.105985 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:11.176790 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:11.176766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:21.136130 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:21.136070 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:21.248081 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:21.248048 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:21.346557 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:21.346523 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:23.500580 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:23.500540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:24.089480 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:24.089443 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:24.187914 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:24.187883 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:26.053059 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:26.053017 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:26.463531 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:26.463495 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:26.563684 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:26.563651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:32.231694 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:32.231650 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:32.332508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:32.332470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:42.623768 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:42.623710 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:42.680148 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:42.680120 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:43.221619 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:43.221583 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:43.278514 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:43.278477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:47.046582 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:47.046545 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:47.149910 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:47.149883 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:49.227955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:49.227915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:49.289408 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:49.289361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:49.554545 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:49.554449 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:49.743481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:49.743419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:51.241206 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.241173 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:51.251352 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251326 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:37:51.251631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251620 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="util" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251633 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="util" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251643 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="pull" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251648 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="pull" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251655 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="pull" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251661 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="pull" Jul 28 10:37:51.251675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251676 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251681 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251688 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="pull" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251692 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="pull" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251701 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251706 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251711 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251716 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251724 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251729 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251752 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251757 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251762 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251767 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="util" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251772 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251777 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251782 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="pull" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251787 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="pull" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251837 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83ae250-366c-4b93-ba4d-843c38ac2eb7" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251846 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f011ac-ad41-4f85-ab22-b760f188eb9b" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251852 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="936f30cc-3c31-4250-9f79-3996a708ed83" containerName="extract" Jul 28 10:37:51.251878 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.251858 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ed5711-83b2-4d9c-96e9-f637d213f034" containerName="extract" Jul 28 10:37:51.254723 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.254708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.257394 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.257373 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 10:37:51.257508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.257409 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 10:37:51.258487 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.258470 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f66n2\"" Jul 28 10:37:51.258577 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.258473 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 10:37:51.262471 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.262448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:37:51.345347 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.345315 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:51.346114 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.346089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:37:51.352470 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.352444 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qht2p\" (UniqueName: \"kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.352542 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.352504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.448937 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.448892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:51.453505 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.453479 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qht2p\" (UniqueName: \"kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.453636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.453541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.454209 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.454189 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.464786 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.464767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qht2p\" (UniqueName: \"kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p\") pod \"limitador-limitador-64c94fd64f-f9zf5\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.566055 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.565998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:51.694286 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:51.694263 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:37:51.695893 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:37:51.695865 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fdd9e9_729d_4bca_b10f_61981a98a55c.slice/crio-939549dc26c361165212e87919b9c9338efeb14b18f98fd79cd86a007b3062bf WatchSource:0}: Error finding container 939549dc26c361165212e87919b9c9338efeb14b18f98fd79cd86a007b3062bf: Status 404 returned error can't find the container with id 939549dc26c361165212e87919b9c9338efeb14b18f98fd79cd86a007b3062bf Jul 28 10:37:52.063317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:52.063271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:52.150627 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:52.150592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:37:52.233402 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:52.233370 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" event={"ID":"a2fdd9e9-729d-4bca-b10f-61981a98a55c","Type":"ContainerStarted","Data":"939549dc26c361165212e87919b9c9338efeb14b18f98fd79cd86a007b3062bf"} Jul 28 10:37:55.250410 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:55.250371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" event={"ID":"a2fdd9e9-729d-4bca-b10f-61981a98a55c","Type":"ContainerStarted","Data":"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d"} Jul 28 10:37:55.250942 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:55.250477 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:37:55.266561 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:37:55.266511 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" podStartSLOduration=1.688986333 podStartE2EDuration="4.266497073s" totalImagesPullingTime="2.57751074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:55.26587198 +0000 UTC m=+872.398531260" watchObservedRunningTime="2026-07-28 10:37:55.266497073 +0000 UTC m=+872.399156352" Jul 28 10:38:06.257375 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.257345 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:38:06.327646 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.327603 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:38:06.327899 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.327859 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" podUID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" containerName="limitador" containerID="cri-o://2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d" gracePeriod=30 Jul 28 10:38:06.874041 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.874015 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:38:06.984565 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.984535 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file\") pod \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " Jul 28 10:38:06.984725 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.984606 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qht2p\" (UniqueName: \"kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p\") pod \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\" (UID: \"a2fdd9e9-729d-4bca-b10f-61981a98a55c\") " Jul 28 10:38:06.984921 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.984894 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file" (OuterVolumeSpecName: "config-file") pod "a2fdd9e9-729d-4bca-b10f-61981a98a55c" (UID: "a2fdd9e9-729d-4bca-b10f-61981a98a55c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:38:06.986811 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:06.986782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p" (OuterVolumeSpecName: "kube-api-access-qht2p") pod "a2fdd9e9-729d-4bca-b10f-61981a98a55c" (UID: "a2fdd9e9-729d-4bca-b10f-61981a98a55c"). InnerVolumeSpecName "kube-api-access-qht2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:38:07.085931 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.085902 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qht2p\" (UniqueName: \"kubernetes.io/projected/a2fdd9e9-729d-4bca-b10f-61981a98a55c-kube-api-access-qht2p\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:38:07.085931 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.085926 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a2fdd9e9-729d-4bca-b10f-61981a98a55c-config-file\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:38:07.312447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.312415 2567 generic.go:356] "Generic (PLEG): container finished" podID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" containerID="2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d" exitCode=0 Jul 28 10:38:07.312886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.312460 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" Jul 28 10:38:07.312886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.312501 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" event={"ID":"a2fdd9e9-729d-4bca-b10f-61981a98a55c","Type":"ContainerDied","Data":"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d"} Jul 28 10:38:07.312886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.312596 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f9zf5" event={"ID":"a2fdd9e9-729d-4bca-b10f-61981a98a55c","Type":"ContainerDied","Data":"939549dc26c361165212e87919b9c9338efeb14b18f98fd79cd86a007b3062bf"} Jul 28 10:38:07.312886 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.312632 2567 scope.go:117] "RemoveContainer" containerID="2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d" Jul 28 10:38:07.321650 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.321633 2567 scope.go:117] "RemoveContainer" containerID="2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d" Jul 28 10:38:07.321923 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:38:07.321906 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d\": container with ID starting with 2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d not found: ID does not exist" containerID="2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d" Jul 28 10:38:07.321982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.321930 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d"} err="failed to get container status \"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d\": rpc error: code = NotFound desc = could not find container \"2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d\": container with ID starting with 2114f3aa70ec555f212b863d7965d21bc5c51ca6b96fc5a975b1f8a54a0fa44d not found: ID does not exist" Jul 28 10:38:07.335462 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.335437 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:38:07.337262 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.337243 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f9zf5"] Jul 28 10:38:07.386380 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.386352 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" path="/var/lib/kubelet/pods/a2fdd9e9-729d-4bca-b10f-61981a98a55c/volumes" Jul 28 10:38:07.397383 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.397348 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:07.521692 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.521607 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:07.882076 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.882005 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs"] Jul 28 10:38:07.882334 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.882321 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" containerName="limitador" Jul 28 10:38:07.882376 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.882338 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" containerName="limitador" Jul 28 10:38:07.882485 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.882472 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2fdd9e9-729d-4bca-b10f-61981a98a55c" containerName="limitador" Jul 28 10:38:07.886819 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.886799 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:07.889750 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.889710 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 28 10:38:07.889876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.889766 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4zg2j\"" Jul 28 10:38:07.889876 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.889798 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 10:38:07.890926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.890908 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 10:38:07.894566 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.894548 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs"] Jul 28 10:38:07.915183 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.915149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:07.994211 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.994182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-data\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:07.994358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:07.994220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2zq\" (UniqueName: \"kubernetes.io/projected/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-kube-api-access-7l2zq\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.095495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.095464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-data\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.095631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.095496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2zq\" (UniqueName: \"kubernetes.io/projected/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-kube-api-access-7l2zq\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.095856 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.095839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-data\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.104529 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.104502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2zq\" (UniqueName: \"kubernetes.io/projected/9a2ce82a-d33d-4a36-8810-3b4e56e56b87-kube-api-access-7l2zq\") pod \"postgres-69fb4594cf-lbbqs\" (UID: \"9a2ce82a-d33d-4a36-8810-3b4e56e56b87\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.198592 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.198525 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:08.329462 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:08.329438 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs"] Jul 28 10:38:08.331578 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:38:08.331557 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2ce82a_d33d_4a36_8810_3b4e56e56b87.slice/crio-65cc3df0234c0e6dfa1b14d3121f436c304230ef67b3fbf97354d8acf8552a33 WatchSource:0}: Error finding container 65cc3df0234c0e6dfa1b14d3121f436c304230ef67b3fbf97354d8acf8552a33: Status 404 returned error can't find the container with id 65cc3df0234c0e6dfa1b14d3121f436c304230ef67b3fbf97354d8acf8552a33 Jul 28 10:38:09.328463 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:09.328384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" event={"ID":"9a2ce82a-d33d-4a36-8810-3b4e56e56b87","Type":"ContainerStarted","Data":"65cc3df0234c0e6dfa1b14d3121f436c304230ef67b3fbf97354d8acf8552a33"} Jul 28 10:38:13.354166 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:13.354132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" event={"ID":"9a2ce82a-d33d-4a36-8810-3b4e56e56b87","Type":"ContainerStarted","Data":"59c9a6ccc26d31935223f94632f274732145fd2ca4d3b1f88dff22ef088fe61b"} Jul 28 10:38:13.354592 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:13.354284 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:13.373980 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:13.373931 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" podStartSLOduration=1.53979665 podStartE2EDuration="6.373917911s" totalImagesPullingTime="4.834121261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:13.372253784 +0000 UTC m=+890.504913060" watchObservedRunningTime="2026-07-28 10:38:13.373917911 +0000 UTC m=+890.506577190" Jul 28 10:38:19.388009 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:19.387934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbbqs" Jul 28 10:38:19.707770 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:19.707703 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:20.122250 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:20.122139 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:27.056817 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.056778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:27.078990 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.078961 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:27.102544 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.101779 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:27.102544 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.101942 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.107935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.107829 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rktk6\"" Jul 28 10:38:27.107935 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.107829 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 10:38:27.109085 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.109068 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 10:38:27.149101 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.149076 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:27.154080 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.154061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8c7p\" (UniqueName: \"kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.154161 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.154094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.206066 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.206034 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:27.254760 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.254711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8c7p\" (UniqueName: \"kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.254872 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.254773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.257537 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.257516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.263302 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.263279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8c7p\" (UniqueName: \"kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p\") pod \"maas-controller-bbd85b7c4-4kcw2\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.415521 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.415464 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:27.542615 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:27.542587 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:27.543863 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:38:27.543837 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eec1438_ea34_42f9_a01b_ce4b790b9b8c.slice/crio-bfce5a8ba89965db06792bb629b7fe20cd800f9090f06633ccefb7fe93e28bc3 WatchSource:0}: Error finding container bfce5a8ba89965db06792bb629b7fe20cd800f9090f06633ccefb7fe93e28bc3: Status 404 returned error can't find the container with id bfce5a8ba89965db06792bb629b7fe20cd800f9090f06633ccefb7fe93e28bc3 Jul 28 10:38:28.429684 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:28.429637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" event={"ID":"1eec1438-ea34-42f9-a01b-ce4b790b9b8c","Type":"ContainerStarted","Data":"bfce5a8ba89965db06792bb629b7fe20cd800f9090f06633ccefb7fe93e28bc3"} Jul 28 10:38:30.024508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:30.024472 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:30.121679 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:30.121606 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:38:30.446309 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:30.446232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" event={"ID":"1eec1438-ea34-42f9-a01b-ce4b790b9b8c","Type":"ContainerStarted","Data":"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af"} Jul 28 10:38:30.446445 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:30.446330 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:30.470807 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:30.470765 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" podStartSLOduration=1.169648427 podStartE2EDuration="3.470728263s" totalImagesPullingTime="2.301079836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:30.470011587 +0000 UTC m=+907.602670868" watchObservedRunningTime="2026-07-28 10:38:30.470728263 +0000 UTC m=+907.603387542" Jul 28 10:38:41.409744 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.409701 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:41.410154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.409885 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" podUID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" containerName="manager" containerID="cri-o://70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af" gracePeriod=30 Jul 28 10:38:41.411710 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.411686 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:41.649479 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.649457 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:41.651258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.651237 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8c7p\" (UniqueName: \"kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p\") pod \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " Jul 28 10:38:41.651345 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.651288 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert\") pod \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\" (UID: \"1eec1438-ea34-42f9-a01b-ce4b790b9b8c\") " Jul 28 10:38:41.653428 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.653393 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1eec1438-ea34-42f9-a01b-ce4b790b9b8c" (UID: "1eec1438-ea34-42f9-a01b-ce4b790b9b8c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:38:41.653523 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.653478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p" (OuterVolumeSpecName: "kube-api-access-w8c7p") pod "1eec1438-ea34-42f9-a01b-ce4b790b9b8c" (UID: "1eec1438-ea34-42f9-a01b-ce4b790b9b8c"). InnerVolumeSpecName "kube-api-access-w8c7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:38:41.751825 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.751803 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-webhook-cert\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:38:41.751825 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:41.751827 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8c7p\" (UniqueName: \"kubernetes.io/projected/1eec1438-ea34-42f9-a01b-ce4b790b9b8c-kube-api-access-w8c7p\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:38:42.505508 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.505474 2567 generic.go:356] "Generic (PLEG): container finished" podID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" containerID="70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af" exitCode=0 Jul 28 10:38:42.505906 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.505513 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" Jul 28 10:38:42.505906 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.505555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" event={"ID":"1eec1438-ea34-42f9-a01b-ce4b790b9b8c","Type":"ContainerDied","Data":"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af"} Jul 28 10:38:42.505906 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.505604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bbd85b7c4-4kcw2" event={"ID":"1eec1438-ea34-42f9-a01b-ce4b790b9b8c","Type":"ContainerDied","Data":"bfce5a8ba89965db06792bb629b7fe20cd800f9090f06633ccefb7fe93e28bc3"} Jul 28 10:38:42.505906 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.505628 2567 scope.go:117] "RemoveContainer" containerID="70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af" Jul 28 10:38:42.514636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.514621 2567 scope.go:117] "RemoveContainer" containerID="70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af" Jul 28 10:38:42.514949 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:38:42.514930 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af\": container with ID starting with 70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af not found: ID does not exist" containerID="70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af" Jul 28 10:38:42.515036 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.514954 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af"} err="failed to get container status \"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af\": rpc error: code = NotFound desc = could not find container \"70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af\": container with ID starting with 70eceb01885a556509179b16a0fdc1a51d327a7430a343ba5cea41173e1f40af not found: ID does not exist" Jul 28 10:38:42.527007 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.526986 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:42.536283 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:42.536262 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bbd85b7c4-4kcw2"] Jul 28 10:38:43.387638 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:38:43.387596 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" path="/var/lib/kubelet/pods/1eec1438-ea34-42f9-a01b-ce4b790b9b8c/volumes" Jul 28 10:39:01.717527 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:01.717484 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:02.453286 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.453257 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf"] Jul 28 10:39:02.453583 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.453570 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" containerName="manager" Jul 28 10:39:02.453631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.453585 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" containerName="manager" Jul 28 10:39:02.453673 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.453642 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eec1438-ea34-42f9-a01b-ce4b790b9b8c" containerName="manager" Jul 28 10:39:02.461639 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.461620 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.465234 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.465215 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 28 10:39:02.465329 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.465311 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m8h9w\"" Jul 28 10:39:02.470394 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.470372 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf"] Jul 28 10:39:02.511063 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.511038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qw5\" (UniqueName: \"kubernetes.io/projected/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-kube-api-access-w8qw5\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.511172 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.511101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.511172 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.511146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-openshift-service-ca\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.612231 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.612204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qw5\" (UniqueName: \"kubernetes.io/projected/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-kube-api-access-w8qw5\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.612350 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.612257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.612350 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.612302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-openshift-service-ca\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.612440 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:39:02.612424 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 28 10:39:02.612518 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:39:02.612506 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls podName:2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9 nodeName:}" failed. No retries permitted until 2026-07-28 10:39:03.112484935 +0000 UTC m=+940.245144192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls") pod "maas-api-77c87dc85b-7zvrf" (UID: "2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9") : secret "maas-api-serving-cert" not found Jul 28 10:39:02.612983 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.612964 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-openshift-service-ca\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:02.621641 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:02.621621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qw5\" (UniqueName: \"kubernetes.io/projected/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-kube-api-access-w8qw5\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:03.057642 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.057610 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq"] Jul 28 10:39:03.068762 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.068712 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.070668 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.070632 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq"] Jul 28 10:39:03.071636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.071612 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 10:39:03.071900 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.071885 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mc5gp\"" Jul 28 10:39:03.103683 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.103650 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ll5xj"] Jul 28 10:39:03.107497 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.107477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.116926 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.116900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lr6\" (UniqueName: \"kubernetes.io/projected/2171fb23-48ec-41f9-a64d-a56007c59f83-kube-api-access-48lr6\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.117041 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.116987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:03.117041 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.117026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2171fb23-48ec-41f9-a64d-a56007c59f83-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.118544 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.118520 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ll5xj"] Jul 28 10:39:03.119823 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.119803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9-maas-api-tls\") pod \"maas-api-77c87dc85b-7zvrf\" (UID: \"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9\") " pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:03.136822 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.136793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:03.218104 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.218075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hqc\" (UniqueName: \"kubernetes.io/projected/e5e93679-4dc9-470d-89c9-a4ba29777ac7-kube-api-access-g8hqc\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.218246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.218118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2171fb23-48ec-41f9-a64d-a56007c59f83-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.218246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.218231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lr6\" (UniqueName: \"kubernetes.io/projected/2171fb23-48ec-41f9-a64d-a56007c59f83-kube-api-access-48lr6\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.218364 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.218308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e93679-4dc9-470d-89c9-a4ba29777ac7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.218746 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.218714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2171fb23-48ec-41f9-a64d-a56007c59f83-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.226987 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.226970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lr6\" (UniqueName: \"kubernetes.io/projected/2171fb23-48ec-41f9-a64d-a56007c59f83-kube-api-access-48lr6\") pod \"payload-pre-processing-687dbb6cf6-v57vq\" (UID: \"2171fb23-48ec-41f9-a64d-a56007c59f83\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.319184 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.319127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hqc\" (UniqueName: \"kubernetes.io/projected/e5e93679-4dc9-470d-89c9-a4ba29777ac7-kube-api-access-g8hqc\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.319278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.319212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e93679-4dc9-470d-89c9-a4ba29777ac7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.319821 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.319803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e93679-4dc9-470d-89c9-a4ba29777ac7-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.329945 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.329917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hqc\" (UniqueName: \"kubernetes.io/projected/e5e93679-4dc9-470d-89c9-a4ba29777ac7-kube-api-access-g8hqc\") pod \"payload-processing-5cc6cd444b-ll5xj\" (UID: \"e5e93679-4dc9-470d-89c9-a4ba29777ac7\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.374320 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.374299 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:03.380194 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.380176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:03.417585 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.417551 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:03.531072 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.531044 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf"] Jul 28 10:39:03.532355 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:39:03.532324 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8e01ef_9e9c_45da_a673_acbeeaa7b2c9.slice/crio-90fd866a1701c1ea1f03d7734b1440b2ba94afcff431c9ad54336a5542d5947b WatchSource:0}: Error finding container 90fd866a1701c1ea1f03d7734b1440b2ba94afcff431c9ad54336a5542d5947b: Status 404 returned error can't find the container with id 90fd866a1701c1ea1f03d7734b1440b2ba94afcff431c9ad54336a5542d5947b Jul 28 10:39:03.533837 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.533816 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:39:03.554947 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.554923 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq"] Jul 28 10:39:03.555712 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:39:03.555683 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2171fb23_48ec_41f9_a64d_a56007c59f83.slice/crio-6d34a6b65367dfa86f7bee4a0b96d6c19dfbb7dd895a4fcfc8f16287a69da9f6 WatchSource:0}: Error finding container 6d34a6b65367dfa86f7bee4a0b96d6c19dfbb7dd895a4fcfc8f16287a69da9f6: Status 404 returned error can't find the container with id 6d34a6b65367dfa86f7bee4a0b96d6c19dfbb7dd895a4fcfc8f16287a69da9f6 Jul 28 10:39:03.581797 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.581775 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ll5xj"] Jul 28 10:39:03.583050 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:39:03.583026 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e93679_4dc9_470d_89c9_a4ba29777ac7.slice/crio-9f1d6dc97fbf3c63b7b312c08de845d964ca446232a59244576244796e4138ad WatchSource:0}: Error finding container 9f1d6dc97fbf3c63b7b312c08de845d964ca446232a59244576244796e4138ad: Status 404 returned error can't find the container with id 9f1d6dc97fbf3c63b7b312c08de845d964ca446232a59244576244796e4138ad Jul 28 10:39:03.614252 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.614222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" event={"ID":"e5e93679-4dc9-470d-89c9-a4ba29777ac7","Type":"ContainerStarted","Data":"9f1d6dc97fbf3c63b7b312c08de845d964ca446232a59244576244796e4138ad"} Jul 28 10:39:03.615480 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.615457 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" event={"ID":"2171fb23-48ec-41f9-a64d-a56007c59f83","Type":"ContainerStarted","Data":"6d34a6b65367dfa86f7bee4a0b96d6c19dfbb7dd895a4fcfc8f16287a69da9f6"} Jul 28 10:39:03.616591 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:03.616567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" event={"ID":"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9","Type":"ContainerStarted","Data":"90fd866a1701c1ea1f03d7734b1440b2ba94afcff431c9ad54336a5542d5947b"} Jul 28 10:39:06.640879 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.640834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" event={"ID":"e5e93679-4dc9-470d-89c9-a4ba29777ac7","Type":"ContainerStarted","Data":"15c9ecf3af919bb1165ff912ebaa3857ea8ef800d704148c822358f080d8e58d"} Jul 28 10:39:06.641349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.640949 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:06.642392 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.642368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" event={"ID":"2171fb23-48ec-41f9-a64d-a56007c59f83","Type":"ContainerStarted","Data":"20665894281edeee04f71bbe8539379b687c7c0ba2c1e53e8b2c0b16afe00838"} Jul 28 10:39:06.642530 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.642513 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:06.643786 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.643765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" event={"ID":"2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9","Type":"ContainerStarted","Data":"47faf6e6fc671b37a0ac9ab46c1a518fa49d5e3d99fca9559fa7ad49cbf57210"} Jul 28 10:39:06.643912 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.643896 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:06.656913 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.656873 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" podStartSLOduration=1.623139559 podStartE2EDuration="3.656860188s" totalImagesPullingTime="2.033720629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:06.65656277 +0000 UTC m=+943.789222049" watchObservedRunningTime="2026-07-28 10:39:06.656860188 +0000 UTC m=+943.789519488" Jul 28 10:39:06.673819 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.673780 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" podStartSLOduration=2.599495493 podStartE2EDuration="4.673768486s" totalImagesPullingTime="2.074272993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:06.671632686 +0000 UTC m=+943.804291968" watchObservedRunningTime="2026-07-28 10:39:06.673768486 +0000 UTC m=+943.806427765" Jul 28 10:39:06.688577 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:06.688536 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" podStartSLOduration=1.638383502 podStartE2EDuration="3.688523619s" totalImagesPullingTime="2.050140117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:06.686180786 +0000 UTC m=+943.818840089" watchObservedRunningTime="2026-07-28 10:39:06.688523619 +0000 UTC m=+943.821182898" Jul 28 10:39:12.655509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:12.655481 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77c87dc85b-7zvrf" Jul 28 10:39:13.679036 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:13.678992 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:13.779042 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:13.778997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:17.652155 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:17.652119 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ll5xj" Jul 28 10:39:17.652612 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:17.652239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v57vq" Jul 28 10:39:18.845021 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.844985 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx"] Jul 28 10:39:18.848783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.848760 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.851441 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.851416 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-65s9d\"" Jul 28 10:39:18.852572 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.852557 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 10:39:18.852666 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.852563 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 10:39:18.852666 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.852584 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 10:39:18.860110 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.860087 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx"] Jul 28 10:39:18.872242 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.872214 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:18.948287 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.948406 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.948406 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bqs\" (UniqueName: \"kubernetes.io/projected/8dae8d03-a402-49d1-b9d9-813a8e49420f-kube-api-access-p9bqs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.948533 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dae8d03-a402-49d1-b9d9-813a8e49420f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.948533 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948498 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:18.948617 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:18.948530 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049245 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bqs\" (UniqueName: \"kubernetes.io/projected/8dae8d03-a402-49d1-b9d9-813a8e49420f-kube-api-access-p9bqs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dae8d03-a402-49d1-b9d9-813a8e49420f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049494 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049494 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049698 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049698 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.049922 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.049811 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.051545 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.051527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dae8d03-a402-49d1-b9d9-813a8e49420f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.051978 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.051955 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dae8d03-a402-49d1-b9d9-813a8e49420f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.057075 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.057054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bqs\" (UniqueName: \"kubernetes.io/projected/8dae8d03-a402-49d1-b9d9-813a8e49420f-kube-api-access-p9bqs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx\" (UID: \"8dae8d03-a402-49d1-b9d9-813a8e49420f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.159573 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.159516 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:19.293035 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.293008 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx"] Jul 28 10:39:19.294710 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:39:19.294684 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dae8d03_a402_49d1_b9d9_813a8e49420f.slice/crio-57085b60ef99778b60de0583e9ceba2adaa6b812aa0ed18c962adfc50f75f98c WatchSource:0}: Error finding container 57085b60ef99778b60de0583e9ceba2adaa6b812aa0ed18c962adfc50f75f98c: Status 404 returned error can't find the container with id 57085b60ef99778b60de0583e9ceba2adaa6b812aa0ed18c962adfc50f75f98c Jul 28 10:39:19.717211 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:19.717172 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" event={"ID":"8dae8d03-a402-49d1-b9d9-813a8e49420f","Type":"ContainerStarted","Data":"57085b60ef99778b60de0583e9ceba2adaa6b812aa0ed18c962adfc50f75f98c"} Jul 28 10:39:20.723674 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:20.723637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" event={"ID":"8dae8d03-a402-49d1-b9d9-813a8e49420f","Type":"ContainerStarted","Data":"cd5d851adcdc12f05c9b3ab6e7ee51689afc45fb8f8956c8e9c669eed7711f86"} Jul 28 10:39:20.741971 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:20.741926 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" podStartSLOduration=1.666669972 podStartE2EDuration="2.74190717s" totalImagesPullingTime="1.075237198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:20.74136749 +0000 UTC m=+957.874026782" watchObservedRunningTime="2026-07-28 10:39:20.74190717 +0000 UTC m=+957.874566449" Jul 28 10:39:24.750338 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:24.750305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:29.159632 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:29.159597 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:29.160246 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:29.159704 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:29.172243 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:29.172216 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:29.783359 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:29.783323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx" Jul 28 10:39:30.269952 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:30.269911 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:41.138996 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:41.138952 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:42.105291 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:42.105253 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:45.860621 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:45.860530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:45.961781 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:45.961745 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:51.150145 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.150112 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f"] Jul 28 10:39:51.154975 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.154950 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.157685 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.157662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 10:39:51.171822 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.171796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f"] Jul 28 10:39:51.201506 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.201456 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:39:51.307430 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307398 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nsf\" (UniqueName: \"kubernetes.io/projected/13a14c06-f593-49f5-99b4-7357bdd738c6-kube-api-access-c5nsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.307430 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.307617 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.307617 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.307729 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.307729 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.307646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a14c06-f593-49f5-99b4-7357bdd738c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nsf\" (UniqueName: \"kubernetes.io/projected/13a14c06-f593-49f5-99b4-7357bdd738c6-kube-api-access-c5nsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408117 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408108 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408344 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a14c06-f593-49f5-99b4-7357bdd738c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408641 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408760 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.408760 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.408665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.410577 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.410556 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a14c06-f593-49f5-99b4-7357bdd738c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.410850 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.410834 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a14c06-f593-49f5-99b4-7357bdd738c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.417720 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.417699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nsf\" (UniqueName: \"kubernetes.io/projected/13a14c06-f593-49f5-99b4-7357bdd738c6-kube-api-access-c5nsf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpm9f\" (UID: \"13a14c06-f593-49f5-99b4-7357bdd738c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.466503 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.466479 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:39:51.600059 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.599621 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f"] Jul 28 10:39:51.608222 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:39:51.608195 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a14c06_f593_49f5_99b4_7357bdd738c6.slice/crio-af81ca4dced9df288f6c094c7166945d5fba5e965fdc5da551e5d22ae1b21068 WatchSource:0}: Error finding container af81ca4dced9df288f6c094c7166945d5fba5e965fdc5da551e5d22ae1b21068: Status 404 returned error can't find the container with id af81ca4dced9df288f6c094c7166945d5fba5e965fdc5da551e5d22ae1b21068 Jul 28 10:39:51.887714 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.887682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" event={"ID":"13a14c06-f593-49f5-99b4-7357bdd738c6","Type":"ContainerStarted","Data":"0942195c9be51cb54c2e1b8ad1b9f30b3984b1e871fbecd0844e827e8c56751b"} Jul 28 10:39:51.887899 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.887772 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" event={"ID":"13a14c06-f593-49f5-99b4-7357bdd738c6","Type":"ContainerStarted","Data":"af81ca4dced9df288f6c094c7166945d5fba5e965fdc5da551e5d22ae1b21068"} Jul 28 10:39:51.908042 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:51.907997 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" podStartSLOduration=0.734658221 podStartE2EDuration="907.983665ms" totalImagesPullingTime="173.325444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:51.90749828 +0000 UTC m=+989.040157560" watchObservedRunningTime="2026-07-28 10:39:51.907983665 +0000 UTC m=+989.040642943" Jul 28 10:39:58.160779 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:39:58.160714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:40:01.467644 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:01.467606 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:40:01.468253 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:01.467674 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:40:01.479912 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:01.479891 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:40:01.958208 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:01.958182 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpm9f" Jul 28 10:40:12.185007 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:12.184970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:40:57.728978 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:57.728937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:40:57.829176 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:57.829136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:40:58.876372 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:58.876336 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:40:58.970332 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:40:58.970292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:41:12.935158 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:41:12.935121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:04.632987 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:04.632907 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:19.265577 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:19.265546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:20.232327 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.232261 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:20.734895 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.734850 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:20.857953 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.857923 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-67vlx"] Jul 28 10:43:20.861287 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.861270 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:20.864157 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.864136 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 10:43:20.865038 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.865019 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rktk6\"" Jul 28 10:43:20.865102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.865085 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 10:43:20.872681 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.872659 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-67vlx"] Jul 28 10:43:20.977619 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.977587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f933be8-6456-4d3f-928c-af17892ee591-webhook-cert\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:20.977779 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:20.977683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvfj\" (UniqueName: \"kubernetes.io/projected/9f933be8-6456-4d3f-928c-af17892ee591-kube-api-access-gvvfj\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.078940 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.078865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvfj\" (UniqueName: \"kubernetes.io/projected/9f933be8-6456-4d3f-928c-af17892ee591-kube-api-access-gvvfj\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.079084 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.078951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f933be8-6456-4d3f-928c-af17892ee591-webhook-cert\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.081729 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.081704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f933be8-6456-4d3f-928c-af17892ee591-webhook-cert\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.087740 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.087718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvfj\" (UniqueName: \"kubernetes.io/projected/9f933be8-6456-4d3f-928c-af17892ee591-kube-api-access-gvvfj\") pod \"maas-controller-84d7d4f6df-67vlx\" (UID: \"9f933be8-6456-4d3f-928c-af17892ee591\") " pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.173044 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.173014 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:21.307109 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:21.307080 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-67vlx"] Jul 28 10:43:21.308619 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:43:21.308593 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f933be8_6456_4d3f_928c_af17892ee591.slice/crio-72132ab8fc375463f127dd5a64bfbf6eebabbfef62cfab47d98257538d6c3f9d WatchSource:0}: Error finding container 72132ab8fc375463f127dd5a64bfbf6eebabbfef62cfab47d98257538d6c3f9d: Status 404 returned error can't find the container with id 72132ab8fc375463f127dd5a64bfbf6eebabbfef62cfab47d98257538d6c3f9d Jul 28 10:43:22.007847 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:22.007813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" event={"ID":"9f933be8-6456-4d3f-928c-af17892ee591","Type":"ContainerStarted","Data":"8e421d8aa6684cb312367cb7ccb1ef634c53388dd1e3f1e03aedc64fee7ff243"} Jul 28 10:43:22.008220 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:22.007863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" event={"ID":"9f933be8-6456-4d3f-928c-af17892ee591","Type":"ContainerStarted","Data":"72132ab8fc375463f127dd5a64bfbf6eebabbfef62cfab47d98257538d6c3f9d"} Jul 28 10:43:22.008220 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:22.007927 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:22.027523 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:22.027470 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" podStartSLOduration=1.677885316 podStartE2EDuration="2.027457859s" totalImagesPullingTime="349.572543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:22.025204583 +0000 UTC m=+1199.157863863" watchObservedRunningTime="2026-07-28 10:43:22.027457859 +0000 UTC m=+1199.160117138" Jul 28 10:43:32.418566 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:32.418529 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:32.478263 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:32.478231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:33.016056 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:33.016019 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d7d4f6df-67vlx" Jul 28 10:43:55.646746 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:55.646712 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:55.739071 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:55.739042 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:43:55.860770 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:43:55.860710 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:01.705481 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:01.705444 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:01.821132 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:01.821029 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:01.971129 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:01.971060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:02.090317 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:02.090286 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:07.848303 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:07.848254 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:44:09.138677 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:44:09.138617 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:45:52.007252 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:45:52.007151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:48:08.194235 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:48:08.194193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:48:08.296646 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:48:08.296605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:50:00.749439 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:50:00.749374 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:50:02.489700 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:50:02.489665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:51:37.211942 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:51:37.211904 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:52:18.762156 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:52:18.762034 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:52:58.588439 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:52:58.588399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:53:39.142367 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:53:39.142335 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:53:40.883390 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:53:40.883351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:56:21.138835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:56:21.138717 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:56:22.995976 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:56:22.995944 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:56:57.770278 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:56:57.770240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:22.152697 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:22.152659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:25.657999 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:25.657964 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:25.830258 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:25.830224 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:26.167257 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:26.167220 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:32.632365 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:32.632328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:32.719925 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:32.719891 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:32.929416 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:32.929335 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:33.308254 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:33.308226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:33.697355 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:33.697271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.247870 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.247833 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.365180 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.365148 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.533588 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.533506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.647341 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.647307 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:57:35.656295 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.656262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.658643 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.658620 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.664073 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.664052 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3eefdb25-serving-cert\"" Jul 28 10:57:35.670648 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.670626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:57:35.756041 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.756015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.756168 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.756067 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.756168 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.756110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b98\" (UniqueName: \"kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.760729 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.760704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.857311 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.857243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.857416 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.857321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.857416 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.857376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b98\" (UniqueName: \"kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.857982 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.857958 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.859901 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.859883 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.870286 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.870261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b98\" (UniqueName: \"kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98\") pod \"maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:35.902226 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.902206 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:35.970026 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:35.970001 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:36.108059 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.107998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:57:36.109164 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:57:36.109137 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1167df_f0aa_459e_9d26_c457b51d57e6.slice/crio-3c519368e50521ab2ab3c1e87ad2858b83d3e8af41a1d3526c3631a85b3ce737 WatchSource:0}: Error finding container 3c519368e50521ab2ab3c1e87ad2858b83d3e8af41a1d3526c3631a85b3ce737: Status 404 returned error can't find the container with id 3c519368e50521ab2ab3c1e87ad2858b83d3e8af41a1d3526c3631a85b3ce737 Jul 28 10:57:36.110470 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.110450 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:57:36.387838 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.387764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:36.473021 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.472987 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" event={"ID":"2e1167df-f0aa-459e-9d26-c457b51d57e6","Type":"ContainerStarted","Data":"3c519368e50521ab2ab3c1e87ad2858b83d3e8af41a1d3526c3631a85b3ce737"} Jul 28 10:57:36.485661 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.485635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:57:36.490434 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.490408 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:57:36.495189 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.495170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.498300 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.498274 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3eefdb25-dockercfg-lqn49\"" Jul 28 10:57:36.498694 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.498679 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3eefdb25\"" Jul 28 10:57:36.518326 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.518303 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:57:36.562963 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.562933 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.563066 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.563025 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfb9\" (UniqueName: \"kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.664350 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.664276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.664350 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.664336 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfb9\" (UniqueName: \"kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.665095 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.665076 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.673835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.673813 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfb9\" (UniqueName: \"kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9\") pod \"payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.805852 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.805828 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:36.945438 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:36.945414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:57:36.947155 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:57:36.947113 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220017f8_567d_4413_b292_7c808b6965d2.slice/crio-eee8d964cffbd93fac76474af5e2228e1786095b6981f7766bcf1843b7989410 WatchSource:0}: Error finding container eee8d964cffbd93fac76474af5e2228e1786095b6981f7766bcf1843b7989410: Status 404 returned error can't find the container with id eee8d964cffbd93fac76474af5e2228e1786095b6981f7766bcf1843b7989410 Jul 28 10:57:37.480390 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.480347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" event={"ID":"2e1167df-f0aa-459e-9d26-c457b51d57e6","Type":"ContainerStarted","Data":"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1"} Jul 28 10:57:37.480881 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.480418 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:37.482168 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.482145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" event={"ID":"220017f8-567d-4413-b292-7c808b6965d2","Type":"ContainerStarted","Data":"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac"} Jul 28 10:57:37.482289 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.482185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" event={"ID":"220017f8-567d-4413-b292-7c808b6965d2","Type":"ContainerStarted","Data":"eee8d964cffbd93fac76474af5e2228e1786095b6981f7766bcf1843b7989410"} Jul 28 10:57:37.482289 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.482205 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:57:37.499223 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.499179 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" podStartSLOduration=1.847810297 podStartE2EDuration="2.499164093s" totalImagesPullingTime="651.353796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:57:37.496505705 +0000 UTC m=+2054.629165012" watchObservedRunningTime="2026-07-28 10:57:37.499164093 +0000 UTC m=+2054.631823407" Jul 28 10:57:37.515114 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:37.515073 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" podStartSLOduration=1.515056205 podStartE2EDuration="1.515056205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:57:37.513425228 +0000 UTC m=+2054.646084508" watchObservedRunningTime="2026-07-28 10:57:37.515056205 +0000 UTC m=+2054.647715484" Jul 28 10:57:43.493282 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:43.493248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:57:48.490784 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:57:48.490680 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:58:11.633352 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.633311 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:58:11.633865 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.633602 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" podUID="2e1167df-f0aa-459e-9d26-c457b51d57e6" containerName="maas-api" containerID="cri-o://bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1" gracePeriod=30 Jul 28 10:58:11.659583 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.659559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:11.691885 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.691858 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:58:11.692284 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.692112 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" podUID="220017f8-567d-4413-b292-7c808b6965d2" containerName="payload-pre-processing" containerID="cri-o://8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac" gracePeriod=30 Jul 28 10:58:11.760203 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.760169 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:11.814453 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.814425 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:11.948030 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.947753 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:58:11.951805 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:11.951786 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:58:12.066636 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.066606 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca\") pod \"2e1167df-f0aa-459e-9d26-c457b51d57e6\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " Jul 28 10:58:12.066842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.066647 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls\") pod \"2e1167df-f0aa-459e-9d26-c457b51d57e6\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " Jul 28 10:58:12.066842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.066667 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5b98\" (UniqueName: \"kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98\") pod \"2e1167df-f0aa-459e-9d26-c457b51d57e6\" (UID: \"2e1167df-f0aa-459e-9d26-c457b51d57e6\") " Jul 28 10:58:12.066842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.066730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume\") pod \"220017f8-567d-4413-b292-7c808b6965d2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " Jul 28 10:58:12.066842 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.066779 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgfb9\" (UniqueName: \"kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9\") pod \"220017f8-567d-4413-b292-7c808b6965d2\" (UID: \"220017f8-567d-4413-b292-7c808b6965d2\") " Jul 28 10:58:12.067104 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.067075 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e1167df-f0aa-459e-9d26-c457b51d57e6" (UID: "2e1167df-f0aa-459e-9d26-c457b51d57e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:58:12.067168 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.067145 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "220017f8-567d-4413-b292-7c808b6965d2" (UID: "220017f8-567d-4413-b292-7c808b6965d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:58:12.069170 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.069143 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9" (OuterVolumeSpecName: "kube-api-access-wgfb9") pod "220017f8-567d-4413-b292-7c808b6965d2" (UID: "220017f8-567d-4413-b292-7c808b6965d2"). InnerVolumeSpecName "kube-api-access-wgfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:58:12.069285 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.069203 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98" (OuterVolumeSpecName: "kube-api-access-h5b98") pod "2e1167df-f0aa-459e-9d26-c457b51d57e6" (UID: "2e1167df-f0aa-459e-9d26-c457b51d57e6"). InnerVolumeSpecName "kube-api-access-h5b98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:58:12.069329 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.069276 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e1167df-f0aa-459e-9d26-c457b51d57e6" (UID: "2e1167df-f0aa-459e-9d26-c457b51d57e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:58:12.167819 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.167795 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220017f8-567d-4413-b292-7c808b6965d2-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:58:12.167819 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.167817 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgfb9\" (UniqueName: \"kubernetes.io/projected/220017f8-567d-4413-b292-7c808b6965d2-kube-api-access-wgfb9\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:58:12.167955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.167827 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1167df-f0aa-459e-9d26-c457b51d57e6-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:58:12.167955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.167836 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1167df-f0aa-459e-9d26-c457b51d57e6-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:58:12.167955 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.167845 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5b98\" (UniqueName: \"kubernetes.io/projected/2e1167df-f0aa-459e-9d26-c457b51d57e6-kube-api-access-h5b98\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:58:12.665854 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.665820 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e1167df-f0aa-459e-9d26-c457b51d57e6" containerID="bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1" exitCode=0 Jul 28 10:58:12.666275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.665876 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" Jul 28 10:58:12.666275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.665890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" event={"ID":"2e1167df-f0aa-459e-9d26-c457b51d57e6","Type":"ContainerDied","Data":"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1"} Jul 28 10:58:12.666275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.665943 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4" event={"ID":"2e1167df-f0aa-459e-9d26-c457b51d57e6","Type":"ContainerDied","Data":"3c519368e50521ab2ab3c1e87ad2858b83d3e8af41a1d3526c3631a85b3ce737"} Jul 28 10:58:12.666275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.665975 2567 scope.go:117] "RemoveContainer" containerID="bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1" Jul 28 10:58:12.667273 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.667253 2567 generic.go:356] "Generic (PLEG): container finished" podID="220017f8-567d-4413-b292-7c808b6965d2" containerID="8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac" exitCode=0 Jul 28 10:58:12.667347 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.667298 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" Jul 28 10:58:12.667347 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.667320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" event={"ID":"220017f8-567d-4413-b292-7c808b6965d2","Type":"ContainerDied","Data":"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac"} Jul 28 10:58:12.667447 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.667359 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2" event={"ID":"220017f8-567d-4413-b292-7c808b6965d2","Type":"ContainerDied","Data":"eee8d964cffbd93fac76474af5e2228e1786095b6981f7766bcf1843b7989410"} Jul 28 10:58:12.675374 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.675359 2567 scope.go:117] "RemoveContainer" containerID="bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1" Jul 28 10:58:12.675624 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:58:12.675608 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1\": container with ID starting with bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1 not found: ID does not exist" containerID="bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1" Jul 28 10:58:12.675675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.675630 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1"} err="failed to get container status \"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1\": rpc error: code = NotFound desc = could not find container \"bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1\": container with ID starting with bef166fbef8ccefeb8a5ef97f4566ba6ddc909a1939e3a53090fca4b65fbb2c1 not found: ID does not exist" Jul 28 10:58:12.675675 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.675647 2567 scope.go:117] "RemoveContainer" containerID="8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac" Jul 28 10:58:12.683875 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.683859 2567 scope.go:117] "RemoveContainer" containerID="8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac" Jul 28 10:58:12.684113 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:58:12.684095 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac\": container with ID starting with 8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac not found: ID does not exist" containerID="8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac" Jul 28 10:58:12.684174 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.684121 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac"} err="failed to get container status \"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac\": rpc error: code = NotFound desc = could not find container \"8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac\": container with ID starting with 8e4c89515e5226f04811aa5e38e375e86fd56629ed91af0df4ae664ede2e92ac not found: ID does not exist" Jul 28 10:58:12.693231 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.693212 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:58:12.695355 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.695333 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3eefdb25-688c6c59dc-jbxc4"] Jul 28 10:58:12.705501 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.705478 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:58:12.709231 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:12.709212 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eefdb25-d4649f456-w2km2"] Jul 28 10:58:13.387280 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:13.387249 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220017f8-567d-4413-b292-7c808b6965d2" path="/var/lib/kubelet/pods/220017f8-567d-4413-b292-7c808b6965d2/volumes" Jul 28 10:58:13.387624 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:13.387610 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1167df-f0aa-459e-9d26-c457b51d57e6" path="/var/lib/kubelet/pods/2e1167df-f0aa-459e-9d26-c457b51d57e6/volumes" Jul 28 10:58:16.307228 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:16.307191 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:16.479163 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:16.479113 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:16.726853 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:16.726823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:16.999356 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:16.999271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:17.106942 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:17.106911 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:17.349699 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:17.349611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:17.739152 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:17.739115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:18.053938 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:18.053855 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.262097 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.262048 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.335181 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335150 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:58:33.335517 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335503 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220017f8-567d-4413-b292-7c808b6965d2" containerName="payload-pre-processing" Jul 28 10:58:33.335517 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335517 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="220017f8-567d-4413-b292-7c808b6965d2" containerName="payload-pre-processing" Jul 28 10:58:33.335669 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335531 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1167df-f0aa-459e-9d26-c457b51d57e6" containerName="maas-api" Jul 28 10:58:33.335669 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335537 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1167df-f0aa-459e-9d26-c457b51d57e6" containerName="maas-api" Jul 28 10:58:33.335669 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335609 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1167df-f0aa-459e-9d26-c457b51d57e6" containerName="maas-api" Jul 28 10:58:33.335669 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.335617 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="220017f8-567d-4413-b292-7c808b6965d2" containerName="payload-pre-processing" Jul 28 10:58:33.339136 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.339097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.342211 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.342177 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fde3f8de-serving-cert\"" Jul 28 10:58:33.349534 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.349509 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:58:33.363805 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.363785 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.468090 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.468055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.536368 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.536291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.536495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.536365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.536495 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.536459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24plr\" (UniqueName: \"kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.637752 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.637713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24plr\" (UniqueName: \"kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.637866 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.637772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.637866 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.637831 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.638482 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.638456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.640548 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.640527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.646911 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.646889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24plr\" (UniqueName: \"kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr\") pod \"maas-api-e2e-migrate-fde3f8de-754478d545-7kf67\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.656584 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.656563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:33.857622 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.857542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.958783 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.958721 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:33.967934 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.967910 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:58:33.971574 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.971551 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:33.975838 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.975821 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fde3f8de-dockercfg-brz9g\"" Jul 28 10:58:33.976033 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.976013 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fde3f8de\"" Jul 28 10:58:33.988184 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:33.988161 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:58:34.011320 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:58:34.011298 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode767d6a7_5a84_44df_9a49_ba382044e70f.slice/crio-b43ac46415b83320e909c4c30273c79004e022f44c47d35d7b10b2f9136780dd WatchSource:0}: Error finding container b43ac46415b83320e909c4c30273c79004e022f44c47d35d7b10b2f9136780dd: Status 404 returned error can't find the container with id b43ac46415b83320e909c4c30273c79004e022f44c47d35d7b10b2f9136780dd Jul 28 10:58:34.011427 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.011324 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:58:34.041816 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.041792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2q8\" (UniqueName: \"kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.041944 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.041852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.142854 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.142767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2q8\" (UniqueName: \"kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.142854 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.142842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.143642 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.143620 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.151554 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.151530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2q8\" (UniqueName: \"kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8\") pod \"payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.283603 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.283573 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.426624 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.426598 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:58:34.428015 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:58:34.427989 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba76e9d_15bd_4227_a926_6bed81909dc6.slice/crio-95ffbb8eb9cc6000d25572a94b00301d805ee204497557437d34e827c5bd5d69 WatchSource:0}: Error finding container 95ffbb8eb9cc6000d25572a94b00301d805ee204497557437d34e827c5bd5d69: Status 404 returned error can't find the container with id 95ffbb8eb9cc6000d25572a94b00301d805ee204497557437d34e827c5bd5d69 Jul 28 10:58:34.784396 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.784356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" event={"ID":"2ba76e9d-15bd-4227-a926-6bed81909dc6","Type":"ContainerStarted","Data":"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef"} Jul 28 10:58:34.784586 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.784430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" event={"ID":"2ba76e9d-15bd-4227-a926-6bed81909dc6","Type":"ContainerStarted","Data":"95ffbb8eb9cc6000d25572a94b00301d805ee204497557437d34e827c5bd5d69"} Jul 28 10:58:34.784586 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.784480 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:58:34.785865 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.785843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" event={"ID":"e767d6a7-5a84-44df-9a49-ba382044e70f","Type":"ContainerStarted","Data":"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f"} Jul 28 10:58:34.785976 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.785878 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" event={"ID":"e767d6a7-5a84-44df-9a49-ba382044e70f","Type":"ContainerStarted","Data":"b43ac46415b83320e909c4c30273c79004e022f44c47d35d7b10b2f9136780dd"} Jul 28 10:58:34.785976 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.785961 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:34.801552 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.801509 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" podStartSLOduration=1.8014965360000001 podStartE2EDuration="1.801496536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:58:34.800708264 +0000 UTC m=+2111.933367544" watchObservedRunningTime="2026-07-28 10:58:34.801496536 +0000 UTC m=+2111.934155824" Jul 28 10:58:34.817392 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:34.817351 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" podStartSLOduration=1.470723774 podStartE2EDuration="1.817336346s" totalImagesPullingTime="346.612572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:58:34.816458134 +0000 UTC m=+2111.949117414" watchObservedRunningTime="2026-07-28 10:58:34.817336346 +0000 UTC m=+2111.949995624" Jul 28 10:58:40.221345 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:40.221306 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:58:40.797253 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:40.797223 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:58:45.793242 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:58:45.793213 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:59:08.874176 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:08.874144 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:59:08.874764 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:08.874370 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" podUID="e767d6a7-5a84-44df-9a49-ba382044e70f" containerName="maas-api" containerID="cri-o://7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f" gracePeriod=30 Jul 28 10:59:08.905038 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:08.905009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:08.944455 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:08.944428 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:59:08.944885 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:08.944844 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" podUID="2ba76e9d-15bd-4227-a926-6bed81909dc6" containerName="payload-pre-processing" containerID="cri-o://59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef" gracePeriod=30 Jul 28 10:59:09.005017 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.004984 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:09.106477 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.106439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:09.230129 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.230108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:59:09.233045 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.233025 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:59:09.322124 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322097 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca\") pod \"e767d6a7-5a84-44df-9a49-ba382044e70f\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " Jul 28 10:59:09.322124 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322128 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz2q8\" (UniqueName: \"kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8\") pod \"2ba76e9d-15bd-4227-a926-6bed81909dc6\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " Jul 28 10:59:09.322353 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322148 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume\") pod \"2ba76e9d-15bd-4227-a926-6bed81909dc6\" (UID: \"2ba76e9d-15bd-4227-a926-6bed81909dc6\") " Jul 28 10:59:09.322353 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322211 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24plr\" (UniqueName: \"kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr\") pod \"e767d6a7-5a84-44df-9a49-ba382044e70f\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " Jul 28 10:59:09.322353 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls\") pod \"e767d6a7-5a84-44df-9a49-ba382044e70f\" (UID: \"e767d6a7-5a84-44df-9a49-ba382044e70f\") " Jul 28 10:59:09.322538 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322510 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e767d6a7-5a84-44df-9a49-ba382044e70f" (UID: "e767d6a7-5a84-44df-9a49-ba382044e70f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:59:09.322607 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.322569 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ba76e9d-15bd-4227-a926-6bed81909dc6" (UID: "2ba76e9d-15bd-4227-a926-6bed81909dc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:59:09.324468 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.324439 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e767d6a7-5a84-44df-9a49-ba382044e70f" (UID: "e767d6a7-5a84-44df-9a49-ba382044e70f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:59:09.324570 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.324514 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8" (OuterVolumeSpecName: "kube-api-access-zz2q8") pod "2ba76e9d-15bd-4227-a926-6bed81909dc6" (UID: "2ba76e9d-15bd-4227-a926-6bed81909dc6"). InnerVolumeSpecName "kube-api-access-zz2q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:59:09.324570 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.324541 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr" (OuterVolumeSpecName: "kube-api-access-24plr") pod "e767d6a7-5a84-44df-9a49-ba382044e70f" (UID: "e767d6a7-5a84-44df-9a49-ba382044e70f"). InnerVolumeSpecName "kube-api-access-24plr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:59:09.423663 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.423633 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24plr\" (UniqueName: \"kubernetes.io/projected/e767d6a7-5a84-44df-9a49-ba382044e70f-kube-api-access-24plr\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:09.423663 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.423660 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e767d6a7-5a84-44df-9a49-ba382044e70f-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:09.423835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.423674 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e767d6a7-5a84-44df-9a49-ba382044e70f-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:09.423835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.423689 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2q8\" (UniqueName: \"kubernetes.io/projected/2ba76e9d-15bd-4227-a926-6bed81909dc6-kube-api-access-zz2q8\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:09.423835 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.423700 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba76e9d-15bd-4227-a926-6bed81909dc6-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:09.994152 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.994117 2567 generic.go:356] "Generic (PLEG): container finished" podID="2ba76e9d-15bd-4227-a926-6bed81909dc6" containerID="59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef" exitCode=0 Jul 28 10:59:09.994551 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.994170 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" Jul 28 10:59:09.994551 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.994199 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" event={"ID":"2ba76e9d-15bd-4227-a926-6bed81909dc6","Type":"ContainerDied","Data":"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef"} Jul 28 10:59:09.994551 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.994290 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48" event={"ID":"2ba76e9d-15bd-4227-a926-6bed81909dc6","Type":"ContainerDied","Data":"95ffbb8eb9cc6000d25572a94b00301d805ee204497557437d34e827c5bd5d69"} Jul 28 10:59:09.994551 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.994320 2567 scope.go:117] "RemoveContainer" containerID="59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef" Jul 28 10:59:09.995915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.995803 2567 generic.go:356] "Generic (PLEG): container finished" podID="e767d6a7-5a84-44df-9a49-ba382044e70f" containerID="7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f" exitCode=0 Jul 28 10:59:09.995915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.995835 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" event={"ID":"e767d6a7-5a84-44df-9a49-ba382044e70f","Type":"ContainerDied","Data":"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f"} Jul 28 10:59:09.995915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.995874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" event={"ID":"e767d6a7-5a84-44df-9a49-ba382044e70f","Type":"ContainerDied","Data":"b43ac46415b83320e909c4c30273c79004e022f44c47d35d7b10b2f9136780dd"} Jul 28 10:59:09.995915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:09.995883 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67" Jul 28 10:59:10.003532 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.003515 2567 scope.go:117] "RemoveContainer" containerID="59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef" Jul 28 10:59:10.003799 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:59:10.003778 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef\": container with ID starting with 59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef not found: ID does not exist" containerID="59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef" Jul 28 10:59:10.003858 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.003809 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef"} err="failed to get container status \"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef\": rpc error: code = NotFound desc = could not find container \"59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef\": container with ID starting with 59ad4ed63fc75ca5b5229e273feebf0ceabbf1595d9915483e9bf859d266f9ef not found: ID does not exist" Jul 28 10:59:10.003858 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.003825 2567 scope.go:117] "RemoveContainer" containerID="7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f" Jul 28 10:59:10.011631 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.011616 2567 scope.go:117] "RemoveContainer" containerID="7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f" Jul 28 10:59:10.011908 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:59:10.011887 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f\": container with ID starting with 7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f not found: ID does not exist" containerID="7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f" Jul 28 10:59:10.011975 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.011913 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f"} err="failed to get container status \"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f\": rpc error: code = NotFound desc = could not find container \"7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f\": container with ID starting with 7a9139d74fd282aaf4e3c56ab72fdb60e62ee801f60bca12d4ca864ec377eb1f not found: ID does not exist" Jul 28 10:59:10.019994 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.019973 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:59:10.026266 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.026242 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fde3f8de-8566bcc996-dqm48"] Jul 28 10:59:10.044811 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.044791 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:59:10.048281 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:10.048258 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fde3f8de-754478d545-7kf67"] Jul 28 10:59:11.386358 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:11.386325 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba76e9d-15bd-4227-a926-6bed81909dc6" path="/var/lib/kubelet/pods/2ba76e9d-15bd-4227-a926-6bed81909dc6/volumes" Jul 28 10:59:11.386723 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:11.386695 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e767d6a7-5a84-44df-9a49-ba382044e70f" path="/var/lib/kubelet/pods/e767d6a7-5a84-44df-9a49-ba382044e70f/volumes" Jul 28 10:59:13.432201 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:13.432144 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:13.596032 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:13.595993 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:13.858987 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:13.858949 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:14.100244 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:14.100171 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:14.447154 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:14.447118 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:14.555570 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:14.555543 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:16.960051 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:16.959988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.048178 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048147 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:17.048780 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048757 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e767d6a7-5a84-44df-9a49-ba382044e70f" containerName="maas-api" Jul 28 10:59:17.048780 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048781 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e767d6a7-5a84-44df-9a49-ba382044e70f" containerName="maas-api" Jul 28 10:59:17.048915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048813 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba76e9d-15bd-4227-a926-6bed81909dc6" containerName="payload-pre-processing" Jul 28 10:59:17.048915 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048822 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba76e9d-15bd-4227-a926-6bed81909dc6" containerName="payload-pre-processing" Jul 28 10:59:17.049026 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048942 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba76e9d-15bd-4227-a926-6bed81909dc6" containerName="payload-pre-processing" Jul 28 10:59:17.049026 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.048960 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e767d6a7-5a84-44df-9a49-ba382044e70f" containerName="maas-api" Jul 28 10:59:17.053963 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.053945 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.057596 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.057577 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4217be61-serving-cert\"" Jul 28 10:59:17.069769 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.062392 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.069769 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.067279 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:17.135186 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.135149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.188986 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.188953 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.189106 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.189006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.189106 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.189057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fw8l\" (UniqueName: \"kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.290382 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.290307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.290382 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.290346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fw8l\" (UniqueName: \"kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.290554 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.290457 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.291107 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.291079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.292938 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.292915 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.299433 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.299414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fw8l\" (UniqueName: \"kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l\") pod \"maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.307106 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.307084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.365030 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.365008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:17.507192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.507166 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:17.509183 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:59:17.509159 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029dd6f9_5de2_4735_8a27_cb1c7285ef21.slice/crio-790794a3dd2bd093e732e13638f035731007634108d02595b929d616a4ec4142 WatchSource:0}: Error finding container 790794a3dd2bd093e732e13638f035731007634108d02595b929d616a4ec4142: Status 404 returned error can't find the container with id 790794a3dd2bd093e732e13638f035731007634108d02595b929d616a4ec4142 Jul 28 10:59:17.639197 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.639128 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.730135 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.730105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:17.753275 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.753244 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:17.758013 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.757996 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:17.762682 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.762662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4217be61-dockercfg-4vv8n\"" Jul 28 10:59:17.763129 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.763108 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4217be61\"" Jul 28 10:59:17.770892 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.770869 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:17.897192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.897110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:17.897192 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.897182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpfj\" (UniqueName: \"kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:17.998086 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.998058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpfj\" (UniqueName: \"kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:17.998361 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.998154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:17.998845 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:17.998825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:18.006453 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.006430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpfj\" (UniqueName: \"kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj\") pod \"payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:18.042621 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.042594 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" event={"ID":"029dd6f9-5de2-4735-8a27-cb1c7285ef21","Type":"ContainerStarted","Data":"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c"} Jul 28 10:59:18.042770 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.042650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" event={"ID":"029dd6f9-5de2-4735-8a27-cb1c7285ef21","Type":"ContainerStarted","Data":"790794a3dd2bd093e732e13638f035731007634108d02595b929d616a4ec4142"} Jul 28 10:59:18.042770 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.042685 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:18.062509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.062461 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" podStartSLOduration=0.636995347 podStartE2EDuration="1.062447394s" totalImagesPullingTime="425.452047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:18.060164307 +0000 UTC m=+2155.192823585" watchObservedRunningTime="2026-07-28 10:59:18.062447394 +0000 UTC m=+2155.195106672" Jul 28 10:59:18.069745 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.069711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:18.209502 ip-10-0-130-55 kubenswrapper[2567]: W0728 10:59:18.209453 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72c403c_fc0a_4d4d_9e7d_2411f5f2a3be.slice/crio-7f30871014bbfed1e00c93741c85529547dd1b3fdc882e49c5d8a8b1c94e6d8f WatchSource:0}: Error finding container 7f30871014bbfed1e00c93741c85529547dd1b3fdc882e49c5d8a8b1c94e6d8f: Status 404 returned error can't find the container with id 7f30871014bbfed1e00c93741c85529547dd1b3fdc882e49c5d8a8b1c94e6d8f Jul 28 10:59:18.209756 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:18.209711 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:19.049363 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:19.049324 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" event={"ID":"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be","Type":"ContainerStarted","Data":"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4"} Jul 28 10:59:19.049789 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:19.049385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" event={"ID":"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be","Type":"ContainerStarted","Data":"7f30871014bbfed1e00c93741c85529547dd1b3fdc882e49c5d8a8b1c94e6d8f"} Jul 28 10:59:19.049789 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:19.049548 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:19.070572 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:19.070526 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" podStartSLOduration=2.070513843 podStartE2EDuration="2.070513843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:19.067355838 +0000 UTC m=+2156.200015152" watchObservedRunningTime="2026-07-28 10:59:19.070513843 +0000 UTC m=+2156.203173122" Jul 28 10:59:24.055029 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:24.054999 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:30.056128 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:30.056101 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:53.467963 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.467924 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:53.468560 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.468171 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" podUID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" containerName="maas-api" containerID="cri-o://59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c" gracePeriod=30 Jul 28 10:59:53.490612 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.490587 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:53.524161 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.524133 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:53.524399 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.524349 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" podUID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" containerName="payload-pre-processing" containerID="cri-o://f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4" gracePeriod=30 Jul 28 10:59:53.588308 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.588273 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:53.649705 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.649663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:53.768102 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.768075 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:53.785251 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.785147 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls\") pod \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " Jul 28 10:59:53.785371 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.785274 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fw8l\" (UniqueName: \"kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l\") pod \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " Jul 28 10:59:53.785371 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.785308 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca\") pod \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\" (UID: \"029dd6f9-5de2-4735-8a27-cb1c7285ef21\") " Jul 28 10:59:53.785810 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.785781 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "029dd6f9-5de2-4735-8a27-cb1c7285ef21" (UID: "029dd6f9-5de2-4735-8a27-cb1c7285ef21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:59:53.789588 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.788951 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l" (OuterVolumeSpecName: "kube-api-access-8fw8l") pod "029dd6f9-5de2-4735-8a27-cb1c7285ef21" (UID: "029dd6f9-5de2-4735-8a27-cb1c7285ef21"). InnerVolumeSpecName "kube-api-access-8fw8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:59:53.790045 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.789799 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "029dd6f9-5de2-4735-8a27-cb1c7285ef21" (UID: "029dd6f9-5de2-4735-8a27-cb1c7285ef21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:59:53.796680 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.796655 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:53.886031 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886009 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpfj\" (UniqueName: \"kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj\") pod \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " Jul 28 10:59:53.886171 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886077 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume\") pod \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\" (UID: \"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be\") " Jul 28 10:59:53.886263 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886252 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fw8l\" (UniqueName: \"kubernetes.io/projected/029dd6f9-5de2-4735-8a27-cb1c7285ef21-kube-api-access-8fw8l\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:53.886302 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886265 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/029dd6f9-5de2-4735-8a27-cb1c7285ef21-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:53.886302 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886275 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/029dd6f9-5de2-4735-8a27-cb1c7285ef21-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:53.886516 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.886487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" (UID: "d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:59:53.888267 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.888248 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj" (OuterVolumeSpecName: "kube-api-access-zrpfj") pod "d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" (UID: "d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be"). InnerVolumeSpecName "kube-api-access-zrpfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:59:53.987584 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.987527 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:53.987584 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:53.987548 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpfj\" (UniqueName: \"kubernetes.io/projected/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be-kube-api-access-zrpfj\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 10:59:54.242349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.242271 2567 generic.go:356] "Generic (PLEG): container finished" podID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" containerID="59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c" exitCode=0 Jul 28 10:59:54.242349 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.242318 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" Jul 28 10:59:54.242549 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.242349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" event={"ID":"029dd6f9-5de2-4735-8a27-cb1c7285ef21","Type":"ContainerDied","Data":"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c"} Jul 28 10:59:54.242549 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.242405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms" event={"ID":"029dd6f9-5de2-4735-8a27-cb1c7285ef21","Type":"ContainerDied","Data":"790794a3dd2bd093e732e13638f035731007634108d02595b929d616a4ec4142"} Jul 28 10:59:54.242549 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.242434 2567 scope.go:117] "RemoveContainer" containerID="59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c" Jul 28 10:59:54.244015 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.243992 2567 generic.go:356] "Generic (PLEG): container finished" podID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" containerID="f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4" exitCode=0 Jul 28 10:59:54.244098 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.244032 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" Jul 28 10:59:54.244098 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.244036 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" event={"ID":"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be","Type":"ContainerDied","Data":"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4"} Jul 28 10:59:54.244098 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.244083 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z" event={"ID":"d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be","Type":"ContainerDied","Data":"7f30871014bbfed1e00c93741c85529547dd1b3fdc882e49c5d8a8b1c94e6d8f"} Jul 28 10:59:54.253408 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.253390 2567 scope.go:117] "RemoveContainer" containerID="59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c" Jul 28 10:59:54.253962 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:59:54.253659 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c\": container with ID starting with 59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c not found: ID does not exist" containerID="59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c" Jul 28 10:59:54.253962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.253687 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c"} err="failed to get container status \"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c\": rpc error: code = NotFound desc = could not find container \"59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c\": container with ID starting with 59195000f14563f0c40b10e12de89a78b0a80611e223abb644e7c26865c9599c not found: ID does not exist" Jul 28 10:59:54.253962 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.253705 2567 scope.go:117] "RemoveContainer" containerID="f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4" Jul 28 10:59:54.262571 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.262552 2567 scope.go:117] "RemoveContainer" containerID="f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4" Jul 28 10:59:54.262873 ip-10-0-130-55 kubenswrapper[2567]: E0728 10:59:54.262855 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4\": container with ID starting with f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4 not found: ID does not exist" containerID="f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4" Jul 28 10:59:54.262932 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.262882 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4"} err="failed to get container status \"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4\": rpc error: code = NotFound desc = could not find container \"f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4\": container with ID starting with f86e4739e5da18afbf7a969c399d1281c85c1c38f4351f1a8381ba6d213c41a4 not found: ID does not exist" Jul 28 10:59:54.268509 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.268484 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:54.272665 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.272644 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4217be61-856bfd76d-hxl8z"] Jul 28 10:59:54.283472 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.283445 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:54.288880 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:54.288860 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4217be61-6d7fc49ddf-jcnms"] Jul 28 10:59:55.387022 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:55.386990 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" path="/var/lib/kubelet/pods/029dd6f9-5de2-4735-8a27-cb1c7285ef21/volumes" Jul 28 10:59:55.387384 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:55.387325 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" path="/var/lib/kubelet/pods/d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be/volumes" Jul 28 10:59:58.048929 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:58.048897 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:59.330232 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:59.330196 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:59.561091 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:59.561053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 10:59:59.814973 ip-10-0-130-55 kubenswrapper[2567]: I0728 10:59:59.814922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:00.168130 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:00.168023 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:00.234616 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:00.234587 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:02.551017 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.550962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:02.634486 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.634455 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:02.635069 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635051 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" containerName="payload-pre-processing" Jul 28 11:00:02.635141 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635071 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" containerName="payload-pre-processing" Jul 28 11:00:02.635141 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635081 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" containerName="maas-api" Jul 28 11:00:02.635141 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635088 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" containerName="maas-api" Jul 28 11:00:02.635250 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635154 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="029dd6f9-5de2-4735-8a27-cb1c7285ef21" containerName="maas-api" Jul 28 11:00:02.635250 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.635163 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72c403c-fc0a-4d4d-9e7d-2411f5f2a3be" containerName="payload-pre-processing" Jul 28 11:00:02.638088 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.638072 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.640983 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.640961 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b5796d54-serving-cert\"" Jul 28 11:00:02.655993 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.655969 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:02.741846 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.741820 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:02.757251 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.757218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.757378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.757264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.757378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.757312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzqt\" (UniqueName: \"kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.858139 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.858065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.858139 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.858105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.858139 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.858124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzqt\" (UniqueName: \"kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.858833 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.858814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.860578 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.860556 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.865968 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.865946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzqt\" (UniqueName: \"kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt\") pod \"maas-api-e2e-derive-b5796d54-9566484f8-6knnl\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.950978 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.950936 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:02.976287 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:02.976265 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:03.074143 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.074105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:03.092815 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.092789 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:03.093996 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:00:03.093960 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5079e4b6_ee22_4ea2_bfbe_64b5ee7e14a1.slice/crio-5d0edd9d9d18861c0e132b8015353ef015f962ff2eb28bfbe8e0b2e47484ced6 WatchSource:0}: Error finding container 5d0edd9d9d18861c0e132b8015353ef015f962ff2eb28bfbe8e0b2e47484ced6: Status 404 returned error can't find the container with id 5d0edd9d9d18861c0e132b8015353ef015f962ff2eb28bfbe8e0b2e47484ced6 Jul 28 11:00:03.241265 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.241232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:03.298367 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.298340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" event={"ID":"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1","Type":"ContainerStarted","Data":"5d0edd9d9d18861c0e132b8015353ef015f962ff2eb28bfbe8e0b2e47484ced6"} Jul 28 11:00:03.340213 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.340181 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:03.344825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.344806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.347635 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.347600 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b5796d54-dockercfg-q9fwg\"" Jul 28 11:00:03.347826 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.347802 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b5796d54\"" Jul 28 11:00:03.355912 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.355888 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:03.465125 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.465085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.465263 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.465135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgvk\" (UniqueName: \"kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.492015 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.491938 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:03.566305 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.566280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.566706 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.566312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgvk\" (UniqueName: \"kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.567301 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.567275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.576699 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.576679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgvk\" (UniqueName: \"kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk\") pod \"payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.657460 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.657432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:03.797875 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:03.797846 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:03.799691 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:00:03.799663 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e364fdb_20cb_41ed_a80f_5cf8212ace8f.slice/crio-7a13f75fab4e7a13846f840ab89719c79dfd0063458458f5122d56a1e9bab8d3 WatchSource:0}: Error finding container 7a13f75fab4e7a13846f840ab89719c79dfd0063458458f5122d56a1e9bab8d3: Status 404 returned error can't find the container with id 7a13f75fab4e7a13846f840ab89719c79dfd0063458458f5122d56a1e9bab8d3 Jul 28 11:00:04.305227 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.305195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" event={"ID":"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1","Type":"ContainerStarted","Data":"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9"} Jul 28 11:00:04.305427 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.305326 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:04.306749 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.306708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" event={"ID":"2e364fdb-20cb-41ed-a80f-5cf8212ace8f","Type":"ContainerStarted","Data":"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09"} Jul 28 11:00:04.306911 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.306779 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:04.306911 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.306796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" event={"ID":"2e364fdb-20cb-41ed-a80f-5cf8212ace8f","Type":"ContainerStarted","Data":"7a13f75fab4e7a13846f840ab89719c79dfd0063458458f5122d56a1e9bab8d3"} Jul 28 11:00:04.335963 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.335905 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" podStartSLOduration=1.8645526829999999 podStartE2EDuration="2.335885658s" totalImagesPullingTime="471.332975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:04.334489419 +0000 UTC m=+2201.467148698" watchObservedRunningTime="2026-07-28 11:00:04.335885658 +0000 UTC m=+2201.468544940" Jul 28 11:00:04.361274 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:04.361228 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" podStartSLOduration=1.3612163179999999 podStartE2EDuration="1.361216318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:04.359262379 +0000 UTC m=+2201.491921661" watchObservedRunningTime="2026-07-28 11:00:04.361216318 +0000 UTC m=+2201.493875597" Jul 28 11:00:10.318107 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:10.318078 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:14.581527 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:14.581490 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:15.314573 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:15.314529 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.48:9004: connect: connection refused" Jul 28 11:00:25.313438 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:25.313403 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:37.956987 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:37.956950 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:37.957481 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:37.957202 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" podUID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" containerName="maas-api" containerID="cri-o://9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9" gracePeriod=30 Jul 28 11:00:37.979452 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:37.979416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:38.014996 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.014964 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:38.015249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.015198 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerName="payload-pre-processing" containerID="cri-o://b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09" gracePeriod=30 Jul 28 11:00:38.135222 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.135151 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:38.235980 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.235884 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:38.261940 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.261919 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:38.288749 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.288707 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:38.353074 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353048 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume\") pod \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " Jul 28 11:00:38.353186 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353081 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjzqt\" (UniqueName: \"kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt\") pod \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " Jul 28 11:00:38.353186 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353120 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls\") pod \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " Jul 28 11:00:38.353186 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353175 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tgvk\" (UniqueName: \"kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk\") pod \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\" (UID: \"2e364fdb-20cb-41ed-a80f-5cf8212ace8f\") " Jul 28 11:00:38.353304 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353203 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca\") pod \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\" (UID: \"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1\") " Jul 28 11:00:38.353466 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e364fdb-20cb-41ed-a80f-5cf8212ace8f" (UID: "2e364fdb-20cb-41ed-a80f-5cf8212ace8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:00:38.353658 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.353627 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" (UID: "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:00:38.355461 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.355438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" (UID: "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:00:38.355536 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.355438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk" (OuterVolumeSpecName: "kube-api-access-4tgvk") pod "2e364fdb-20cb-41ed-a80f-5cf8212ace8f" (UID: "2e364fdb-20cb-41ed-a80f-5cf8212ace8f"). InnerVolumeSpecName "kube-api-access-4tgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:00:38.355536 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.355479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt" (OuterVolumeSpecName: "kube-api-access-sjzqt") pod "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" (UID: "5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1"). InnerVolumeSpecName "kube-api-access-sjzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:00:38.453882 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.453850 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgvk\" (UniqueName: \"kubernetes.io/projected/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-kube-api-access-4tgvk\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:00:38.453882 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.453876 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:00:38.453882 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.453886 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e364fdb-20cb-41ed-a80f-5cf8212ace8f-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:00:38.454083 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.453895 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzqt\" (UniqueName: \"kubernetes.io/projected/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-kube-api-access-sjzqt\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:00:38.454083 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.453905 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:00:38.492319 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.492246 2567 generic.go:356] "Generic (PLEG): container finished" podID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" containerID="9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9" exitCode=0 Jul 28 11:00:38.492319 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.492290 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" Jul 28 11:00:38.492496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.492337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" event={"ID":"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1","Type":"ContainerDied","Data":"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9"} Jul 28 11:00:38.492496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.492393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl" event={"ID":"5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1","Type":"ContainerDied","Data":"5d0edd9d9d18861c0e132b8015353ef015f962ff2eb28bfbe8e0b2e47484ced6"} Jul 28 11:00:38.492496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.492418 2567 scope.go:117] "RemoveContainer" containerID="9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9" Jul 28 11:00:38.493805 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.493777 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerID="b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09" exitCode=0 Jul 28 11:00:38.494094 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.493820 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" Jul 28 11:00:38.494094 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.493819 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" event={"ID":"2e364fdb-20cb-41ed-a80f-5cf8212ace8f","Type":"ContainerDied","Data":"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09"} Jul 28 11:00:38.494094 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.493875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292" event={"ID":"2e364fdb-20cb-41ed-a80f-5cf8212ace8f","Type":"ContainerDied","Data":"7a13f75fab4e7a13846f840ab89719c79dfd0063458458f5122d56a1e9bab8d3"} Jul 28 11:00:38.501533 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.501498 2567 scope.go:117] "RemoveContainer" containerID="9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9" Jul 28 11:00:38.501969 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:00:38.501815 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9\": container with ID starting with 9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9 not found: ID does not exist" containerID="9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9" Jul 28 11:00:38.501969 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.501848 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9"} err="failed to get container status \"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9\": rpc error: code = NotFound desc = could not find container \"9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9\": container with ID starting with 9445d31ea35651b80672a07a0cd0b890ccd3fa8748a2025f1f0f9cf712b204f9 not found: ID does not exist" Jul 28 11:00:38.501969 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.501873 2567 scope.go:117] "RemoveContainer" containerID="b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09" Jul 28 11:00:38.510321 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.510306 2567 scope.go:117] "RemoveContainer" containerID="b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09" Jul 28 11:00:38.510546 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:00:38.510526 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09\": container with ID starting with b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09 not found: ID does not exist" containerID="b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09" Jul 28 11:00:38.510609 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.510553 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09"} err="failed to get container status \"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09\": rpc error: code = NotFound desc = could not find container \"b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09\": container with ID starting with b488907e129c3b2692bdf9a05ccad8f66ff69ea8f678b5ad0136abefe9b0be09 not found: ID does not exist" Jul 28 11:00:38.517816 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.517795 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:38.524410 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.524386 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5796d54-cb64c5848-2l292"] Jul 28 11:00:38.535002 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.534984 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:38.537021 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:38.537004 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b5796d54-9566484f8-6knnl"] Jul 28 11:00:39.387081 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:39.387008 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" path="/var/lib/kubelet/pods/2e364fdb-20cb-41ed-a80f-5cf8212ace8f/volumes" Jul 28 11:00:39.387408 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:39.387363 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" path="/var/lib/kubelet/pods/5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1/volumes" Jul 28 11:00:42.644754 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:42.644708 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:42.803824 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:42.803793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:43.064816 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:43.064783 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:43.568056 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:43.568013 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:43.658134 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:43.658101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:43.758198 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:43.758167 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:45.261688 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:45.261614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:51.125078 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:51.125035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:51.239586 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:51.239554 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:51.325805 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:51.325770 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:51.429446 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:51.429375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:00:51.836333 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:00:51.836302 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:18.021385 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:18.021340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:24.253716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:24.253680 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:25.688989 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:25.688945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:40.508365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:40.508333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:46.301891 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:46.301856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:46.366289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:46.366261 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:46.455891 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:46.455861 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:46.556607 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:46.556522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:57.126454 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:57.126411 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:01:58.051086 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:01:58.051048 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:04.604974 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:04.604937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:04.708995 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:04.708966 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:04.805583 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:04.805552 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:04.906607 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:04.906544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:16.584140 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:16.584053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:16.682236 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:16.682204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:16.781700 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:16.781665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:18.612179 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:18.612152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:24.490057 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:24.490026 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:24.590642 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:24.590611 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:25.227886 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:25.227853 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:30.922782 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:30.922718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:30.985414 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:30.985382 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:31.193664 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:31.193584 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:31.580369 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:31.580333 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:31.877058 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:31.876980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:34.932723 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:34.932682 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.017639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.017598 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:35.018150 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018130 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" containerName="maas-api" Jul 28 11:02:35.018150 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018150 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" containerName="maas-api" Jul 28 11:02:35.018344 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018168 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerName="payload-pre-processing" Jul 28 11:02:35.018344 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018174 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerName="payload-pre-processing" Jul 28 11:02:35.018344 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018250 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5079e4b6-ee22-4ea2-bfbe-64b5ee7e14a1" containerName="maas-api" Jul 28 11:02:35.018344 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018259 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e364fdb-20cb-41ed-a80f-5cf8212ace8f" containerName="payload-pre-processing" Jul 28 11:02:35.018995 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.018954 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.021463 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.021446 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.024136 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.024122 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-df73bc23-serving-cert\"" Jul 28 11:02:35.031598 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.031576 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:35.091334 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.091313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.177407 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.177373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.177407 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.177407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.177652 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.177491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mn4\" (UniqueName: \"kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.226112 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.226086 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.278716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.278689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.278922 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.278809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.278922 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.278888 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mn4\" (UniqueName: \"kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.279490 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.279469 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.281453 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.281432 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.287415 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.287395 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mn4\" (UniqueName: \"kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4\") pod \"maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.332309 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.332283 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:35.472019 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.471988 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:35.473390 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:02:35.473367 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e352baa_3030_4e37_b683_4a7bad511a21.slice/crio-96d5bee6a11b293cfca0444b6e605b4095085c9717bfc979f866173c3ea84320 WatchSource:0}: Error finding container 96d5bee6a11b293cfca0444b6e605b4095085c9717bfc979f866173c3ea84320: Status 404 returned error can't find the container with id 96d5bee6a11b293cfca0444b6e605b4095085c9717bfc979f866173c3ea84320 Jul 28 11:02:35.639557 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.639522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.728221 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.728142 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:35.735107 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.735080 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:35.739547 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.739528 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.742350 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.742327 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df73bc23\"" Jul 28 11:02:35.749204 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.749184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:35.884696 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.884664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.884871 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.884769 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44cj\" (UniqueName: \"kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.985530 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.985461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44cj\" (UniqueName: \"kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.985530 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.985521 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.986204 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.986183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:35.994210 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:35.994188 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44cj\" (UniqueName: \"kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj\") pod \"payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:36.052089 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.052068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:36.125949 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.125916 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" event={"ID":"2e352baa-3030-4e37-b683-4a7bad511a21","Type":"ContainerStarted","Data":"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06"} Jul 28 11:02:36.126100 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.126065 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:36.126146 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.126115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" event={"ID":"2e352baa-3030-4e37-b683-4a7bad511a21","Type":"ContainerStarted","Data":"96d5bee6a11b293cfca0444b6e605b4095085c9717bfc979f866173c3ea84320"} Jul 28 11:02:36.147334 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.147098 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" podStartSLOduration=1.589350595 podStartE2EDuration="2.147079495s" totalImagesPullingTime="557.7289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:36.146904056 +0000 UTC m=+2353.279563347" watchObservedRunningTime="2026-07-28 11:02:36.147079495 +0000 UTC m=+2353.279738776" Jul 28 11:02:36.189276 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:36.189250 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:36.190789 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:02:36.190719 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9950ece8_161b_4e9c_a9bc_68fbb09e869c.slice/crio-188789b570f7b34680ffea02bb4fcdf9de2f03fb3f0d668f753336bf1486a532 WatchSource:0}: Error finding container 188789b570f7b34680ffea02bb4fcdf9de2f03fb3f0d668f753336bf1486a532: Status 404 returned error can't find the container with id 188789b570f7b34680ffea02bb4fcdf9de2f03fb3f0d668f753336bf1486a532 Jul 28 11:02:37.132370 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:37.132337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" event={"ID":"9950ece8-161b-4e9c-a9bc-68fbb09e869c","Type":"ContainerStarted","Data":"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2"} Jul 28 11:02:37.132781 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:37.132423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" event={"ID":"9950ece8-161b-4e9c-a9bc-68fbb09e869c","Type":"ContainerStarted","Data":"188789b570f7b34680ffea02bb4fcdf9de2f03fb3f0d668f753336bf1486a532"} Jul 28 11:02:37.132781 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:37.132450 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:37.150070 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:37.150025 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" podStartSLOduration=2.150010887 podStartE2EDuration="2.150010887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:37.148565025 +0000 UTC m=+2354.281224307" watchObservedRunningTime="2026-07-28 11:02:37.150010887 +0000 UTC m=+2354.282670163" Jul 28 11:02:39.477326 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:39.477292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:40.204221 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:40.204174 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:40.295731 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:40.295697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:40.690563 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:40.690529 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:42.138695 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:42.138666 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:46.365937 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:46.365890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:46.426059 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:46.426033 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:48.139600 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:48.139568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:57.179586 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.179547 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:57.180079 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.179825 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" podUID="2e352baa-3030-4e37-b683-4a7bad511a21" containerName="maas-api" containerID="cri-o://a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06" gracePeriod=30 Jul 28 11:02:57.207762 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.207708 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:57.233512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.233485 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:57.233813 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.233763 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" podUID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" containerName="payload-pre-processing" containerID="cri-o://697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2" gracePeriod=30 Jul 28 11:02:57.335201 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.335168 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:57.392406 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.392362 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:02:57.482918 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.482892 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:57.513324 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.513300 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:57.564627 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.564603 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99mn4\" (UniqueName: \"kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4\") pod \"2e352baa-3030-4e37-b683-4a7bad511a21\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " Jul 28 11:02:57.564797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.564656 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g44cj\" (UniqueName: \"kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj\") pod \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " Jul 28 11:02:57.564797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.564729 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca\") pod \"2e352baa-3030-4e37-b683-4a7bad511a21\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " Jul 28 11:02:57.564797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.564787 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls\") pod \"2e352baa-3030-4e37-b683-4a7bad511a21\" (UID: \"2e352baa-3030-4e37-b683-4a7bad511a21\") " Jul 28 11:02:57.564988 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.564811 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume\") pod \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\" (UID: \"9950ece8-161b-4e9c-a9bc-68fbb09e869c\") " Jul 28 11:02:57.565151 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.565126 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e352baa-3030-4e37-b683-4a7bad511a21" (UID: "2e352baa-3030-4e37-b683-4a7bad511a21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:02:57.565214 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.565195 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9950ece8-161b-4e9c-a9bc-68fbb09e869c" (UID: "9950ece8-161b-4e9c-a9bc-68fbb09e869c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:02:57.566914 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.566896 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e352baa-3030-4e37-b683-4a7bad511a21" (UID: "2e352baa-3030-4e37-b683-4a7bad511a21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:02:57.567331 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.567309 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj" (OuterVolumeSpecName: "kube-api-access-g44cj") pod "9950ece8-161b-4e9c-a9bc-68fbb09e869c" (UID: "9950ece8-161b-4e9c-a9bc-68fbb09e869c"). InnerVolumeSpecName "kube-api-access-g44cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:02:57.567377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.567343 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4" (OuterVolumeSpecName: "kube-api-access-99mn4") pod "2e352baa-3030-4e37-b683-4a7bad511a21" (UID: "2e352baa-3030-4e37-b683-4a7bad511a21"). InnerVolumeSpecName "kube-api-access-99mn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:02:57.665583 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.665549 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e352baa-3030-4e37-b683-4a7bad511a21-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:02:57.665583 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.665581 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9950ece8-161b-4e9c-a9bc-68fbb09e869c-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:02:57.665583 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.665591 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99mn4\" (UniqueName: \"kubernetes.io/projected/2e352baa-3030-4e37-b683-4a7bad511a21-kube-api-access-99mn4\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:02:57.665777 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.665600 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44cj\" (UniqueName: \"kubernetes.io/projected/9950ece8-161b-4e9c-a9bc-68fbb09e869c-kube-api-access-g44cj\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:02:57.665777 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:57.665608 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e352baa-3030-4e37-b683-4a7bad511a21-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:02:58.247030 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.247002 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e352baa-3030-4e37-b683-4a7bad511a21" containerID="a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06" exitCode=0 Jul 28 11:02:58.247449 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.247066 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" Jul 28 11:02:58.247449 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.247080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" event={"ID":"2e352baa-3030-4e37-b683-4a7bad511a21","Type":"ContainerDied","Data":"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06"} Jul 28 11:02:58.247449 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.247131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd" event={"ID":"2e352baa-3030-4e37-b683-4a7bad511a21","Type":"ContainerDied","Data":"96d5bee6a11b293cfca0444b6e605b4095085c9717bfc979f866173c3ea84320"} Jul 28 11:02:58.247449 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.247163 2567 scope.go:117] "RemoveContainer" containerID="a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06" Jul 28 11:02:58.248516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.248432 2567 generic.go:356] "Generic (PLEG): container finished" podID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" containerID="697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2" exitCode=0 Jul 28 11:02:58.248516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.248479 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" Jul 28 11:02:58.248516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.248487 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" event={"ID":"9950ece8-161b-4e9c-a9bc-68fbb09e869c","Type":"ContainerDied","Data":"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2"} Jul 28 11:02:58.248691 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.248529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j" event={"ID":"9950ece8-161b-4e9c-a9bc-68fbb09e869c","Type":"ContainerDied","Data":"188789b570f7b34680ffea02bb4fcdf9de2f03fb3f0d668f753336bf1486a532"} Jul 28 11:02:58.257579 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.257562 2567 scope.go:117] "RemoveContainer" containerID="a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06" Jul 28 11:02:58.257933 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:02:58.257909 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06\": container with ID starting with a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06 not found: ID does not exist" containerID="a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06" Jul 28 11:02:58.258002 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.257940 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06"} err="failed to get container status \"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06\": rpc error: code = NotFound desc = could not find container \"a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06\": container with ID starting with a12617bfa818713f431632d7fa4eb30ddc05faeea603441d94a6db193a979f06 not found: ID does not exist" Jul 28 11:02:58.258002 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.257957 2567 scope.go:117] "RemoveContainer" containerID="697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2" Jul 28 11:02:58.266274 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.266258 2567 scope.go:117] "RemoveContainer" containerID="697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2" Jul 28 11:02:58.266540 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:02:58.266523 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2\": container with ID starting with 697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2 not found: ID does not exist" containerID="697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2" Jul 28 11:02:58.266607 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.266548 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2"} err="failed to get container status \"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2\": rpc error: code = NotFound desc = could not find container \"697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2\": container with ID starting with 697b3ce3f145d089015f60a7181f9e9f41a1180309c13d33126fd934e5a58fd2 not found: ID does not exist" Jul 28 11:02:58.273046 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.273025 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:58.276870 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.276848 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df73bc23-56f68455c6-rqj6j"] Jul 28 11:02:58.288059 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.288037 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:58.292062 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:58.292041 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df73bc23-64d69857f8-5r6nd"] Jul 28 11:02:59.385971 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:59.385942 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e352baa-3030-4e37-b683-4a7bad511a21" path="/var/lib/kubelet/pods/2e352baa-3030-4e37-b683-4a7bad511a21/volumes" Jul 28 11:02:59.386316 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:02:59.386289 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" path="/var/lib/kubelet/pods/9950ece8-161b-4e9c-a9bc-68fbb09e869c/volumes" Jul 28 11:03:01.878669 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:01.878637 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:02.319625 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:02.319589 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:08.269281 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:08.269235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:08.339561 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:08.339526 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:08.569833 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:08.569724 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:08.807874 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:08.807842 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:08.950157 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:08.950080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:09.044293 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:09.044265 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:09.189279 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:09.189243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:15.015034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:15.014994 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:15.083810 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:15.083781 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:15.181786 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:15.181728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:15.248314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:15.248284 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:15.362091 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:15.362022 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:21.294425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:21.294387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:21.394626 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:21.394590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:28.752362 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:28.752321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:29.113330 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:29.113230 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:29.176650 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:29.176619 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:32.243927 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.243897 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:32.346984 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.346951 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:32.496372 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.496294 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:03:32.496921 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.496900 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e352baa-3030-4e37-b683-4a7bad511a21" containerName="maas-api" Jul 28 11:03:32.496984 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.496926 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e352baa-3030-4e37-b683-4a7bad511a21" containerName="maas-api" Jul 28 11:03:32.496984 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.496941 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" containerName="payload-pre-processing" Jul 28 11:03:32.496984 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.496950 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" containerName="payload-pre-processing" Jul 28 11:03:32.497087 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.497063 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9950ece8-161b-4e9c-a9bc-68fbb09e869c" containerName="payload-pre-processing" Jul 28 11:03:32.497123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.497093 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e352baa-3030-4e37-b683-4a7bad511a21" containerName="maas-api" Jul 28 11:03:32.500639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.500619 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.504321 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.504295 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0b092d88-serving-cert\"" Jul 28 11:03:32.517716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.517695 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:03:32.556949 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.556921 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:32.645627 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.645589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.645807 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.645638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmgj\" (UniqueName: \"kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.645807 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.645724 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.746464 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.746391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.746464 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.746427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmgj\" (UniqueName: \"kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.746639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.746467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.746639 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:03:32.746583 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-serving-cert: secret "maas-api-e2e-mt-0b092d88-serving-cert" not found Jul 28 11:03:32.746786 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:03:32.746657 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls podName:78f601c8-9731-4789-8b10-d8c2ac24b81b nodeName:}" failed. No retries permitted until 2026-07-28 11:03:33.246639429 +0000 UTC m=+2410.379298686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls") pod "maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" (UID: "78f601c8-9731-4789-8b10-d8c2ac24b81b") : secret "maas-api-e2e-mt-0b092d88-serving-cert" not found Jul 28 11:03:32.747125 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.747105 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.757538 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.757514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmgj\" (UniqueName: \"kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:32.932970 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:32.932942 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:33.240329 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.240301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:33.249417 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.249388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:33.251962 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.251937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") pod \"maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:33.340657 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.340627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:33.401014 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.400992 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:03:33.404578 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.404562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.407705 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.407688 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0b092d88-dockercfg-fb94k\"" Jul 28 11:03:33.407826 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.407715 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0b092d88\"" Jul 28 11:03:33.410427 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.410404 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:33.418665 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.418640 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:03:33.552466 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.552429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c659b\" (UniqueName: \"kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.552641 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.552580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.567368 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.567330 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:03:33.569338 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:03:33.569312 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f601c8_9731_4789_8b10_d8c2ac24b81b.slice/crio-df7fb09830293553e4b283ff92897bb50e20f70ec66ac6b30a1c58fdf368bb86 WatchSource:0}: Error finding container df7fb09830293553e4b283ff92897bb50e20f70ec66ac6b30a1c58fdf368bb86: Status 404 returned error can't find the container with id df7fb09830293553e4b283ff92897bb50e20f70ec66ac6b30a1c58fdf368bb86 Jul 28 11:03:33.571162 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.571143 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 11:03:33.653454 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.653423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c659b\" (UniqueName: \"kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.653584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.653478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.654142 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.654123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.662839 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.662821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c659b\" (UniqueName: \"kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b\") pod \"payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.715504 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.715487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:33.859156 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:33.859133 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:03:33.860727 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:03:33.860697 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158a7a9f_8bca_4fba_be76_490a17702929.slice/crio-0b37bab692c64a09f311f16987b390912fd2a2b79bac1ff2961b4630c0204e6a WatchSource:0}: Error finding container 0b37bab692c64a09f311f16987b390912fd2a2b79bac1ff2961b4630c0204e6a: Status 404 returned error can't find the container with id 0b37bab692c64a09f311f16987b390912fd2a2b79bac1ff2961b4630c0204e6a Jul 28 11:03:34.449880 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.449784 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" event={"ID":"78f601c8-9731-4789-8b10-d8c2ac24b81b","Type":"ContainerStarted","Data":"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876"} Jul 28 11:03:34.449880 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.449849 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:34.449880 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.449872 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" event={"ID":"78f601c8-9731-4789-8b10-d8c2ac24b81b","Type":"ContainerStarted","Data":"df7fb09830293553e4b283ff92897bb50e20f70ec66ac6b30a1c58fdf368bb86"} Jul 28 11:03:34.451302 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.451275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" event={"ID":"158a7a9f-8bca-4fba-be76-490a17702929","Type":"ContainerStarted","Data":"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d"} Jul 28 11:03:34.451443 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.451326 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" event={"ID":"158a7a9f-8bca-4fba-be76-490a17702929","Type":"ContainerStarted","Data":"0b37bab692c64a09f311f16987b390912fd2a2b79bac1ff2961b4630c0204e6a"} Jul 28 11:03:34.451443 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.451371 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:03:34.476101 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.476039 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" podStartSLOduration=1.8745511430000001 podStartE2EDuration="2.476025306s" totalImagesPullingTime="601.474163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:03:34.474369523 +0000 UTC m=+2411.607028802" watchObservedRunningTime="2026-07-28 11:03:34.476025306 +0000 UTC m=+2411.608684585" Jul 28 11:03:34.503437 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:34.503388 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" podStartSLOduration=1.503374699 podStartE2EDuration="1.503374699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:03:34.501799004 +0000 UTC m=+2411.634458284" watchObservedRunningTime="2026-07-28 11:03:34.503374699 +0000 UTC m=+2411.636034013" Jul 28 11:03:36.751947 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:36.751912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:40.462226 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:40.462195 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:03:42.159925 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:42.159896 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:44.585352 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:44.585271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:44.704786 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:44.704755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:03:45.458829 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:03:45.458802 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:04:26.871211 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:26.871176 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:04:26.871848 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:26.871443 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" podUID="78f601c8-9731-4789-8b10-d8c2ac24b81b" containerName="maas-api" containerID="cri-o://4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876" gracePeriod=30 Jul 28 11:04:26.901696 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:26.901670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:04:26.933704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:26.933669 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:04:26.934475 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:26.933981 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" podUID="158a7a9f-8bca-4fba-be76-490a17702929" containerName="payload-pre-processing" containerID="cri-o://c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d" gracePeriod=30 Jul 28 11:04:27.208087 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.208056 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:04:27.211154 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.211124 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:04:27.292234 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292203 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmgj\" (UniqueName: \"kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj\") pod \"78f601c8-9731-4789-8b10-d8c2ac24b81b\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " Jul 28 11:04:27.292377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume\") pod \"158a7a9f-8bca-4fba-be76-490a17702929\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " Jul 28 11:04:27.292377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292285 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca\") pod \"78f601c8-9731-4789-8b10-d8c2ac24b81b\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " Jul 28 11:04:27.292377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292319 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c659b\" (UniqueName: \"kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b\") pod \"158a7a9f-8bca-4fba-be76-490a17702929\" (UID: \"158a7a9f-8bca-4fba-be76-490a17702929\") " Jul 28 11:04:27.292377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292353 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") pod \"78f601c8-9731-4789-8b10-d8c2ac24b81b\" (UID: \"78f601c8-9731-4789-8b10-d8c2ac24b81b\") " Jul 28 11:04:27.292681 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292653 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78f601c8-9731-4789-8b10-d8c2ac24b81b" (UID: "78f601c8-9731-4789-8b10-d8c2ac24b81b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:04:27.292764 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.292682 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "158a7a9f-8bca-4fba-be76-490a17702929" (UID: "158a7a9f-8bca-4fba-be76-490a17702929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:04:27.294512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.294486 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b" (OuterVolumeSpecName: "kube-api-access-c659b") pod "158a7a9f-8bca-4fba-be76-490a17702929" (UID: "158a7a9f-8bca-4fba-be76-490a17702929"). InnerVolumeSpecName "kube-api-access-c659b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:04:27.294512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.294507 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj" (OuterVolumeSpecName: "kube-api-access-5vmgj") pod "78f601c8-9731-4789-8b10-d8c2ac24b81b" (UID: "78f601c8-9731-4789-8b10-d8c2ac24b81b"). InnerVolumeSpecName "kube-api-access-5vmgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:04:27.294645 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.294509 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78f601c8-9731-4789-8b10-d8c2ac24b81b" (UID: "78f601c8-9731-4789-8b10-d8c2ac24b81b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:04:27.339868 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.339841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:04:27.393425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.393377 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c659b\" (UniqueName: \"kubernetes.io/projected/158a7a9f-8bca-4fba-be76-490a17702929-kube-api-access-c659b\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:04:27.393425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.393394 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78f601c8-9731-4789-8b10-d8c2ac24b81b-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:04:27.393425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.393404 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmgj\" (UniqueName: \"kubernetes.io/projected/78f601c8-9731-4789-8b10-d8c2ac24b81b-kube-api-access-5vmgj\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:04:27.393425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.393412 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158a7a9f-8bca-4fba-be76-490a17702929-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:04:27.393425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.393420 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f601c8-9731-4789-8b10-d8c2ac24b81b-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:04:27.397581 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.397559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:04:27.739657 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.739618 2567 generic.go:356] "Generic (PLEG): container finished" podID="78f601c8-9731-4789-8b10-d8c2ac24b81b" containerID="4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876" exitCode=0 Jul 28 11:04:27.739828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.739682 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" Jul 28 11:04:27.739828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.739706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" event={"ID":"78f601c8-9731-4789-8b10-d8c2ac24b81b","Type":"ContainerDied","Data":"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876"} Jul 28 11:04:27.739828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.739810 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq" event={"ID":"78f601c8-9731-4789-8b10-d8c2ac24b81b","Type":"ContainerDied","Data":"df7fb09830293553e4b283ff92897bb50e20f70ec66ac6b30a1c58fdf368bb86"} Jul 28 11:04:27.739971 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.739835 2567 scope.go:117] "RemoveContainer" containerID="4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876" Jul 28 11:04:27.741078 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.741056 2567 generic.go:356] "Generic (PLEG): container finished" podID="158a7a9f-8bca-4fba-be76-490a17702929" containerID="c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d" exitCode=0 Jul 28 11:04:27.741182 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.741132 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" Jul 28 11:04:27.741182 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.741151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" event={"ID":"158a7a9f-8bca-4fba-be76-490a17702929","Type":"ContainerDied","Data":"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d"} Jul 28 11:04:27.741259 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.741195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8" event={"ID":"158a7a9f-8bca-4fba-be76-490a17702929","Type":"ContainerDied","Data":"0b37bab692c64a09f311f16987b390912fd2a2b79bac1ff2961b4630c0204e6a"} Jul 28 11:04:27.749016 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.748999 2567 scope.go:117] "RemoveContainer" containerID="4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876" Jul 28 11:04:27.749272 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:04:27.749254 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876\": container with ID starting with 4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876 not found: ID does not exist" containerID="4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876" Jul 28 11:04:27.749388 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.749283 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876"} err="failed to get container status \"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876\": rpc error: code = NotFound desc = could not find container \"4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876\": container with ID starting with 4565961544b5d751883864a4835b8414ab5ace37d776f932a6b9c4614ee09876 not found: ID does not exist" Jul 28 11:04:27.749388 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.749329 2567 scope.go:117] "RemoveContainer" containerID="c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d" Jul 28 11:04:27.758256 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.758240 2567 scope.go:117] "RemoveContainer" containerID="c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d" Jul 28 11:04:27.758427 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.758407 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:04:27.758530 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:04:27.758513 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d\": container with ID starting with c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d not found: ID does not exist" containerID="c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d" Jul 28 11:04:27.758571 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.758537 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d"} err="failed to get container status \"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d\": rpc error: code = NotFound desc = could not find container \"c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d\": container with ID starting with c35f81fb64ec2c0e4fd0e9c2337961b4344555c44ba6d160221089a97161872d not found: ID does not exist" Jul 28 11:04:27.762355 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.762333 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0b092d88-8c4d4c5b8-tp2m8"] Jul 28 11:04:27.773300 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.773280 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:04:27.776189 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:27.776170 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0b092d88-5f5bd454d5-zmrxq"] Jul 28 11:04:29.387879 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:29.387849 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158a7a9f-8bca-4fba-be76-490a17702929" path="/var/lib/kubelet/pods/158a7a9f-8bca-4fba-be76-490a17702929/volumes" Jul 28 11:04:29.388231 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:29.388205 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f601c8-9731-4789-8b10-d8c2ac24b81b" path="/var/lib/kubelet/pods/78f601c8-9731-4789-8b10-d8c2ac24b81b/volumes" Jul 28 11:04:31.517074 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:31.517044 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:04:34.909114 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:34.909085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:04:39.941356 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:04:39.941323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:16.336044 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:16.335968 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:17.388665 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:17.388628 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:22.838212 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:22.838175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:23.795938 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:23.795853 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:23.896578 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:23.896548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:24.730320 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:24.730283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:24.822161 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:24.822135 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:24.922496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:24.922468 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:25.755061 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:25.755025 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:25.849051 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:25.849019 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:25.948759 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:25.948706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:27.528326 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:27.528291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:33.395325 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:33.395296 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:33.469083 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:33.469036 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:34.148622 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:34.148588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:39.832024 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:39.831977 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:39.934681 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:39.934641 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:40.036455 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:40.036423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:48.022897 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:48.022862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:50.602592 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:50.602545 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:05:50.970013 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:05:50.969977 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:09.688088 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:09.688053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:11.942604 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:11.942567 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:18.214297 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:18.214258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:18.315133 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:18.315101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:18.579718 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:18.579610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:18.905858 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:18.905718 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:19.089167 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:19.089132 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.040355 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.040320 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.138808 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.138766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.258570 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.258537 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:06:22.258967 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.258951 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158a7a9f-8bca-4fba-be76-490a17702929" containerName="payload-pre-processing" Jul 28 11:06:22.258967 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.258966 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="158a7a9f-8bca-4fba-be76-490a17702929" containerName="payload-pre-processing" Jul 28 11:06:22.259136 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.259014 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f601c8-9731-4789-8b10-d8c2ac24b81b" containerName="maas-api" Jul 28 11:06:22.259136 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.259021 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f601c8-9731-4789-8b10-d8c2ac24b81b" containerName="maas-api" Jul 28 11:06:22.259136 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.259075 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f601c8-9731-4789-8b10-d8c2ac24b81b" containerName="maas-api" Jul 28 11:06:22.259136 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.259084 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="158a7a9f-8bca-4fba-be76-490a17702929" containerName="payload-pre-processing" Jul 28 11:06:22.262042 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.262023 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.264642 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.264620 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-730a1a-serving-cert\"" Jul 28 11:06:22.272710 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.272690 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:06:22.292814 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.292757 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.406250 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.406221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.406378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.406253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz2n\" (UniqueName: \"kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.406378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.406352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.507354 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.507322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.507354 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.507352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz2n\" (UniqueName: \"kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.507541 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:06:22.507474 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-serving-cert: secret "maas-api-e2e-inf-a-730a1a-serving-cert" not found Jul 28 11:06:22.507587 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:06:22.507547 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls podName:734040b8-d56b-4534-9106-2f83d31cc998 nodeName:}" failed. No retries permitted until 2026-07-28 11:06:23.007526787 +0000 UTC m=+2580.140186048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls") pod "maas-api-e2e-inf-a-730a1a-646df67445-trgn9" (UID: "734040b8-d56b-4534-9106-2f83d31cc998") : secret "maas-api-e2e-inf-a-730a1a-serving-cert" not found Jul 28 11:06:22.507701 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.507676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.508239 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.508222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.518427 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.518405 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz2n\" (UniqueName: \"kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:22.742933 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.742905 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.841149 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.841115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:22.857044 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.857016 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:06:22.860620 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.860601 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:22.863437 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.863412 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-730a1a\"" Jul 28 11:06:22.863553 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.863470 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-730a1a-dockercfg-h8p42\"" Jul 28 11:06:22.870982 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.870955 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:06:22.918329 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:22.918304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:23.012516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.012432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:23.012653 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.012537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknfl\" (UniqueName: \"kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.012653 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.012564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.015436 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.015410 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") pod \"maas-api-e2e-inf-a-730a1a-646df67445-trgn9\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:23.113772 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.113713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.114193 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.113911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknfl\" (UniqueName: \"kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.114393 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.114372 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.124360 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.124331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknfl\" (UniqueName: \"kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl\") pod \"payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.174939 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.174913 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:23.176338 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.176320 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-730a1a-dockercfg-h8p42\"" Jul 28 11:06:23.184400 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.184380 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:23.325315 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.325285 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:06:23.326912 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:06:23.326878 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734040b8_d56b_4534_9106_2f83d31cc998.slice/crio-3c4d8a82dcef3775562aa1507de6810042e6530de7e6ec5e10951c00b1d5cc58 WatchSource:0}: Error finding container 3c4d8a82dcef3775562aa1507de6810042e6530de7e6ec5e10951c00b1d5cc58: Status 404 returned error can't find the container with id 3c4d8a82dcef3775562aa1507de6810042e6530de7e6ec5e10951c00b1d5cc58 Jul 28 11:06:23.355049 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.354940 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" event={"ID":"734040b8-d56b-4534-9106-2f83d31cc998","Type":"ContainerStarted","Data":"3c4d8a82dcef3775562aa1507de6810042e6530de7e6ec5e10951c00b1d5cc58"} Jul 28 11:06:23.356403 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:23.356379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:06:23.357724 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:06:23.357701 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f8ec45_1b74_4a7f_b788_9f46b2c44b39.slice/crio-05f0b815aef34095a8cbd5a1f4c015f95cbc655524ed7cb1fb88abe863a410ff WatchSource:0}: Error finding container 05f0b815aef34095a8cbd5a1f4c015f95cbc655524ed7cb1fb88abe863a410ff: Status 404 returned error can't find the container with id 05f0b815aef34095a8cbd5a1f4c015f95cbc655524ed7cb1fb88abe863a410ff Jul 28 11:06:24.369344 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.369245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" event={"ID":"734040b8-d56b-4534-9106-2f83d31cc998","Type":"ContainerStarted","Data":"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd"} Jul 28 11:06:24.369838 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.369506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:24.371137 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.371068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" event={"ID":"96f8ec45-1b74-4a7f-b788-9f46b2c44b39","Type":"ContainerStarted","Data":"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb"} Jul 28 11:06:24.371402 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.371297 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" event={"ID":"96f8ec45-1b74-4a7f-b788-9f46b2c44b39","Type":"ContainerStarted","Data":"05f0b815aef34095a8cbd5a1f4c015f95cbc655524ed7cb1fb88abe863a410ff"} Jul 28 11:06:24.371402 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.371356 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:24.386506 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.386463 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" podStartSLOduration=1.73955664 podStartE2EDuration="2.386446582s" totalImagesPullingTime="646.889942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:24.385918806 +0000 UTC m=+2581.518578086" watchObservedRunningTime="2026-07-28 11:06:24.386446582 +0000 UTC m=+2581.519105863" Jul 28 11:06:24.403298 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:24.403262 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" podStartSLOduration=2.403249956 podStartE2EDuration="2.403249956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:24.400918826 +0000 UTC m=+2581.533578105" watchObservedRunningTime="2026-07-28 11:06:24.403249956 +0000 UTC m=+2581.535909252" Jul 28 11:06:26.309541 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:26.309506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:27.411115 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:27.411080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:27.833919 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:27.833883 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:29.732491 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.732460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:29.797846 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.797816 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:06:29.810217 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.810123 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:29.813148 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.813123 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2298f8-serving-cert\"" Jul 28 11:06:29.813603 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.813576 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:06:29.815342 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.815316 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:29.883544 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.883520 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:29.976469 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.976437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8d5\" (UniqueName: \"kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:29.976589 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.976471 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:29.976589 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:29.976489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.077639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.077566 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8d5\" (UniqueName: \"kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.077639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.077597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.077639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.077616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.078277 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.078258 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.080126 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.080104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.090818 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.090728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8d5\" (UniqueName: \"kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5\") pod \"maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.110649 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.110626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:30.123027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.123007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:30.260929 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.260900 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:06:30.263052 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:06:30.263025 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0570dfe3_ae93_4d6a_8463_855ec86e739d.slice/crio-691c008a827b3daf32d81cfa79f42fc929ada6784f3da964a347c45b581919cf WatchSource:0}: Error finding container 691c008a827b3daf32d81cfa79f42fc929ada6784f3da964a347c45b581919cf: Status 404 returned error can't find the container with id 691c008a827b3daf32d81cfa79f42fc929ada6784f3da964a347c45b581919cf Jul 28 11:06:30.381659 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.381593 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:06:30.406697 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.406667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" event={"ID":"0570dfe3-ae93-4d6a-8463-855ec86e739d","Type":"ContainerStarted","Data":"691c008a827b3daf32d81cfa79f42fc929ada6784f3da964a347c45b581919cf"} Jul 28 11:06:30.457029 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.457001 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:30.558706 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:30.558673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:31.413720 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:31.413679 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" event={"ID":"0570dfe3-ae93-4d6a-8463-855ec86e739d","Type":"ContainerStarted","Data":"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9"} Jul 28 11:06:31.414147 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:31.413801 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:31.432039 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:31.431994 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" podStartSLOduration=2.020198328 podStartE2EDuration="2.431979024s" totalImagesPullingTime="411.780696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:31.430110131 +0000 UTC m=+2588.562769412" watchObservedRunningTime="2026-07-28 11:06:31.431979024 +0000 UTC m=+2588.564638302" Jul 28 11:06:34.254478 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:34.254445 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:35.378933 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:35.378896 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:06:37.423855 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:37.423829 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:06:51.013611 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:51.013512 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:06:56.380521 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:06:56.380482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:07:35.594940 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:07:35.594901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:07:57.353812 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:07:57.353710 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:15.165420 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:15.165346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:16.089065 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:16.089035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:26.611690 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.611639 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:08:26.612701 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.612627 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" podUID="734040b8-d56b-4534-9106-2f83d31cc998" containerName="maas-api" containerID="cri-o://e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd" gracePeriod=30 Jul 28 11:08:26.641261 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.641231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:26.669435 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.669397 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:08:26.669706 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.669653 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" podUID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" containerName="payload-pre-processing" containerID="cri-o://d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb" gracePeriod=30 Jul 28 11:08:26.741952 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.741914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:26.793260 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.793223 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:26.918053 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.918029 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:08:26.940079 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.940043 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbz2n\" (UniqueName: \"kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n\") pod \"734040b8-d56b-4534-9106-2f83d31cc998\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " Jul 28 11:08:26.940199 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.940090 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca\") pod \"734040b8-d56b-4534-9106-2f83d31cc998\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " Jul 28 11:08:26.940259 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.940236 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") pod \"734040b8-d56b-4534-9106-2f83d31cc998\" (UID: \"734040b8-d56b-4534-9106-2f83d31cc998\") " Jul 28 11:08:26.940531 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.940504 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "734040b8-d56b-4534-9106-2f83d31cc998" (UID: "734040b8-d56b-4534-9106-2f83d31cc998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:08:26.942335 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.942306 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n" (OuterVolumeSpecName: "kube-api-access-vbz2n") pod "734040b8-d56b-4534-9106-2f83d31cc998" (UID: "734040b8-d56b-4534-9106-2f83d31cc998"). InnerVolumeSpecName "kube-api-access-vbz2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:08:26.942494 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.942469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "734040b8-d56b-4534-9106-2f83d31cc998" (UID: "734040b8-d56b-4534-9106-2f83d31cc998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:08:26.949355 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:26.949335 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:08:27.040820 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.040795 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume\") pod \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " Jul 28 11:08:27.040993 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.040845 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vknfl\" (UniqueName: \"kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl\") pod \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\" (UID: \"96f8ec45-1b74-4a7f-b788-9f46b2c44b39\") " Jul 28 11:08:27.041076 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.041060 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734040b8-d56b-4534-9106-2f83d31cc998-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:27.041123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.041084 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbz2n\" (UniqueName: \"kubernetes.io/projected/734040b8-d56b-4534-9106-2f83d31cc998-kube-api-access-vbz2n\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:27.041123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.041098 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734040b8-d56b-4534-9106-2f83d31cc998-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:27.041207 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.041146 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96f8ec45-1b74-4a7f-b788-9f46b2c44b39" (UID: "96f8ec45-1b74-4a7f-b788-9f46b2c44b39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:08:27.043087 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.043061 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl" (OuterVolumeSpecName: "kube-api-access-vknfl") pod "96f8ec45-1b74-4a7f-b788-9f46b2c44b39" (UID: "96f8ec45-1b74-4a7f-b788-9f46b2c44b39"). InnerVolumeSpecName "kube-api-access-vknfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:08:27.044486 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.044464 2567 generic.go:356] "Generic (PLEG): container finished" podID="734040b8-d56b-4534-9106-2f83d31cc998" containerID="e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd" exitCode=0 Jul 28 11:08:27.044584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.044508 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" Jul 28 11:08:27.044584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.044544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" event={"ID":"734040b8-d56b-4534-9106-2f83d31cc998","Type":"ContainerDied","Data":"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd"} Jul 28 11:08:27.044679 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.044610 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9" event={"ID":"734040b8-d56b-4534-9106-2f83d31cc998","Type":"ContainerDied","Data":"3c4d8a82dcef3775562aa1507de6810042e6530de7e6ec5e10951c00b1d5cc58"} Jul 28 11:08:27.044679 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.044645 2567 scope.go:117] "RemoveContainer" containerID="e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd" Jul 28 11:08:27.045884 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.045866 2567 generic.go:356] "Generic (PLEG): container finished" podID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" containerID="d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb" exitCode=0 Jul 28 11:08:27.045970 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.045909 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" Jul 28 11:08:27.045970 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.045928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" event={"ID":"96f8ec45-1b74-4a7f-b788-9f46b2c44b39","Type":"ContainerDied","Data":"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb"} Jul 28 11:08:27.046042 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.045975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx" event={"ID":"96f8ec45-1b74-4a7f-b788-9f46b2c44b39","Type":"ContainerDied","Data":"05f0b815aef34095a8cbd5a1f4c015f95cbc655524ed7cb1fb88abe863a410ff"} Jul 28 11:08:27.053723 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.053704 2567 scope.go:117] "RemoveContainer" containerID="e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd" Jul 28 11:08:27.053985 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:08:27.053962 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd\": container with ID starting with e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd not found: ID does not exist" containerID="e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd" Jul 28 11:08:27.054061 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.053996 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd"} err="failed to get container status \"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd\": rpc error: code = NotFound desc = could not find container \"e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd\": container with ID starting with e586f5c218ae09e9e6e40461d708c84a4bdddf754c9738d57b3bb8b217e584bd not found: ID does not exist" Jul 28 11:08:27.054061 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.054019 2567 scope.go:117] "RemoveContainer" containerID="d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb" Jul 28 11:08:27.062679 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.062666 2567 scope.go:117] "RemoveContainer" containerID="d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb" Jul 28 11:08:27.064958 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:08:27.064938 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb\": container with ID starting with d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb not found: ID does not exist" containerID="d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb" Jul 28 11:08:27.065043 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.064966 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb"} err="failed to get container status \"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb\": rpc error: code = NotFound desc = could not find container \"d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb\": container with ID starting with d3b978c44614fffc586a0d94129814194947b058a6538b6d9550c3a86e3b02fb not found: ID does not exist" Jul 28 11:08:27.071843 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.071815 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:08:27.075216 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.075196 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-730a1a-7b84b89bd-chsxx"] Jul 28 11:08:27.085113 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.085094 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:08:27.088500 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.088481 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-730a1a-646df67445-trgn9"] Jul 28 11:08:27.141766 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.141686 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vknfl\" (UniqueName: \"kubernetes.io/projected/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-kube-api-access-vknfl\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:27.141766 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.141707 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f8ec45-1b74-4a7f-b788-9f46b2c44b39-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:27.387389 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.387352 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734040b8-d56b-4534-9106-2f83d31cc998" path="/var/lib/kubelet/pods/734040b8-d56b-4534-9106-2f83d31cc998/volumes" Jul 28 11:08:27.387723 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:27.387710 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" path="/var/lib/kubelet/pods/96f8ec45-1b74-4a7f-b788-9f46b2c44b39/volumes" Jul 28 11:08:31.304217 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:31.304180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:31.693729 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:31.693635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:37.566828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:37.566791 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:37.667756 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:37.667709 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:37.799468 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:37.799415 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:38.069937 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:38.069887 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:48.956701 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:48.956666 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:08:48.957625 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:48.957574 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" podUID="0570dfe3-ae93-4d6a-8463-855ec86e739d" containerName="maas-api" containerID="cri-o://2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9" gracePeriod=30 Jul 28 11:08:48.981378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:48.981351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:49.052452 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.052419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:49.112294 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.112268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:49.425083 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.425056 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:08:49.535393 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.535327 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj8d5\" (UniqueName: \"kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5\") pod \"0570dfe3-ae93-4d6a-8463-855ec86e739d\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " Jul 28 11:08:49.535393 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.535356 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls\") pod \"0570dfe3-ae93-4d6a-8463-855ec86e739d\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " Jul 28 11:08:49.535567 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.535413 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca\") pod \"0570dfe3-ae93-4d6a-8463-855ec86e739d\" (UID: \"0570dfe3-ae93-4d6a-8463-855ec86e739d\") " Jul 28 11:08:49.535948 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.535915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0570dfe3-ae93-4d6a-8463-855ec86e739d" (UID: "0570dfe3-ae93-4d6a-8463-855ec86e739d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:08:49.536459 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.536440 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0570dfe3-ae93-4d6a-8463-855ec86e739d-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:49.537686 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.537664 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5" (OuterVolumeSpecName: "kube-api-access-nj8d5") pod "0570dfe3-ae93-4d6a-8463-855ec86e739d" (UID: "0570dfe3-ae93-4d6a-8463-855ec86e739d"). InnerVolumeSpecName "kube-api-access-nj8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:08:49.537806 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.537762 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0570dfe3-ae93-4d6a-8463-855ec86e739d" (UID: "0570dfe3-ae93-4d6a-8463-855ec86e739d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:08:49.637498 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.637476 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8d5\" (UniqueName: \"kubernetes.io/projected/0570dfe3-ae93-4d6a-8463-855ec86e739d-kube-api-access-nj8d5\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:49.637498 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:49.637499 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0570dfe3-ae93-4d6a-8463-855ec86e739d-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:08:50.175023 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.174989 2567 generic.go:356] "Generic (PLEG): container finished" podID="0570dfe3-ae93-4d6a-8463-855ec86e739d" containerID="2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9" exitCode=0 Jul 28 11:08:50.175023 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.175028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" event={"ID":"0570dfe3-ae93-4d6a-8463-855ec86e739d","Type":"ContainerDied","Data":"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9"} Jul 28 11:08:50.175546 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.175034 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" Jul 28 11:08:50.175546 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.175067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm" event={"ID":"0570dfe3-ae93-4d6a-8463-855ec86e739d","Type":"ContainerDied","Data":"691c008a827b3daf32d81cfa79f42fc929ada6784f3da964a347c45b581919cf"} Jul 28 11:08:50.175546 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.175092 2567 scope.go:117] "RemoveContainer" containerID="2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9" Jul 28 11:08:50.185168 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.185151 2567 scope.go:117] "RemoveContainer" containerID="2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9" Jul 28 11:08:50.185432 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:08:50.185408 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9\": container with ID starting with 2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9 not found: ID does not exist" containerID="2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9" Jul 28 11:08:50.185506 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.185440 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9"} err="failed to get container status \"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9\": rpc error: code = NotFound desc = could not find container \"2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9\": container with ID starting with 2006cc1476e4edac2334172a8b9ce6f0418ec5c41f6aba5565d532c0983b9ee9 not found: ID does not exist" Jul 28 11:08:50.196988 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.196960 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:08:50.203438 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:50.203414 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2298f8-5b6f97bcdb-7wnwm"] Jul 28 11:08:51.387557 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:51.387520 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0570dfe3-ae93-4d6a-8463-855ec86e739d" path="/var/lib/kubelet/pods/0570dfe3-ae93-4d6a-8463-855ec86e739d/volumes" Jul 28 11:08:53.275423 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:53.275388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:53.678522 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:53.678433 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:59.530908 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:59.530880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:59.592088 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:59.592061 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:08:59.808335 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:08:59.808260 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:00.034415 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:00.034383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:00.423670 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:00.423633 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:00.776183 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:00.776146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:00.944318 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:00.944262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:03.898204 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:03.898166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.084465 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.084409 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.095814 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.095791 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:04.096225 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096210 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734040b8-d56b-4534-9106-2f83d31cc998" containerName="maas-api" Jul 28 11:09:04.096289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096228 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="734040b8-d56b-4534-9106-2f83d31cc998" containerName="maas-api" Jul 28 11:09:04.096289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096241 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" containerName="payload-pre-processing" Jul 28 11:09:04.096289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096249 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" containerName="payload-pre-processing" Jul 28 11:09:04.096289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096264 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0570dfe3-ae93-4d6a-8463-855ec86e739d" containerName="maas-api" Jul 28 11:09:04.096289 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096273 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0570dfe3-ae93-4d6a-8463-855ec86e739d" containerName="maas-api" Jul 28 11:09:04.096485 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096361 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="734040b8-d56b-4534-9106-2f83d31cc998" containerName="maas-api" Jul 28 11:09:04.096485 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096374 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f8ec45-1b74-4a7f-b788-9f46b2c44b39" containerName="payload-pre-processing" Jul 28 11:09:04.096485 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.096388 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0570dfe3-ae93-4d6a-8463-855ec86e739d" containerName="maas-api" Jul 28 11:09:04.099493 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.099475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.102381 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.102363 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1912e6-serving-cert\"" Jul 28 11:09:04.111371 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.111348 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:04.175062 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.174998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.258472 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.258433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.258472 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.258472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg68w\" (UniqueName: \"kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.258682 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.258506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.359519 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.359488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.359651 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.359520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg68w\" (UniqueName: \"kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.359651 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.359551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.359783 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:04.359672 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-serving-cert: secret "maas-api-e2e-api-a-1912e6-serving-cert" not found Jul 28 11:09:04.359850 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:04.359781 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls podName:a552659c-3442-4c03-b4a2-db032241a747 nodeName:}" failed. No retries permitted until 2026-07-28 11:09:04.85975729 +0000 UTC m=+2741.992416561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls") pod "maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" (UID: "a552659c-3442-4c03-b4a2-db032241a747") : secret "maas-api-e2e-api-a-1912e6-serving-cert" not found Jul 28 11:09:04.360331 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.360301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.370809 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.370788 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg68w\" (UniqueName: \"kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.408782 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.408757 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.762286 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.762252 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.861714 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.861688 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:04.864540 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.864519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.867174 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.867152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") pod \"maas-api-e2e-api-a-1912e6-5b8978d957-sttqc\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:04.870364 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.870340 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:04.874111 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.874091 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:04.876985 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.876965 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1912e6-dockercfg-clbh9\"" Jul 28 11:09:04.877095 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.877003 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1912e6\"" Jul 28 11:09:04.885633 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:04.885613 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:05.010087 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.010055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:05.066439 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.066179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.066439 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.066270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcp9m\" (UniqueName: \"kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.152168 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.152137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:05.153332 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:09:05.153296 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda552659c_3442_4c03_b4a2_db032241a747.slice/crio-65e470cfe04b36dee5e28daea8443c9a772cabb3e9417da701d30a5e99c160ae WatchSource:0}: Error finding container 65e470cfe04b36dee5e28daea8443c9a772cabb3e9417da701d30a5e99c160ae: Status 404 returned error can't find the container with id 65e470cfe04b36dee5e28daea8443c9a772cabb3e9417da701d30a5e99c160ae Jul 28 11:09:05.154684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.154667 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 11:09:05.167704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.167682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcp9m\" (UniqueName: \"kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.167830 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.167806 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.168495 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.168465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.175763 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.175724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcp9m\" (UniqueName: \"kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m\") pod \"payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.185640 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.185617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:05.258286 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.258236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" event={"ID":"a552659c-3442-4c03-b4a2-db032241a747","Type":"ContainerStarted","Data":"65e470cfe04b36dee5e28daea8443c9a772cabb3e9417da701d30a5e99c160ae"} Jul 28 11:09:05.322926 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:05.322892 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:05.324446 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:09:05.324419 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512b7d47_8758_4041_9b4d_f234c7f35929.slice/crio-5f3eb6f2ae32791f6d6d29e39dae0d907a1396a1bebb833c1e352a8128164d05 WatchSource:0}: Error finding container 5f3eb6f2ae32791f6d6d29e39dae0d907a1396a1bebb833c1e352a8128164d05: Status 404 returned error can't find the container with id 5f3eb6f2ae32791f6d6d29e39dae0d907a1396a1bebb833c1e352a8128164d05 Jul 28 11:09:06.265439 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.265399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" event={"ID":"a552659c-3442-4c03-b4a2-db032241a747","Type":"ContainerStarted","Data":"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892"} Jul 28 11:09:06.265900 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.265472 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:06.267135 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.267109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" event={"ID":"512b7d47-8758-4041-9b4d-f234c7f35929","Type":"ContainerStarted","Data":"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd"} Jul 28 11:09:06.267253 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.267153 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:06.267253 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.267170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" event={"ID":"512b7d47-8758-4041-9b4d-f234c7f35929","Type":"ContainerStarted","Data":"5f3eb6f2ae32791f6d6d29e39dae0d907a1396a1bebb833c1e352a8128164d05"} Jul 28 11:09:06.282032 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.281954 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" podStartSLOduration=1.8104767320000001 podStartE2EDuration="2.281935709s" totalImagesPullingTime="471.458977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:06.280929123 +0000 UTC m=+2743.413588390" watchObservedRunningTime="2026-07-28 11:09:06.281935709 +0000 UTC m=+2743.414594991" Jul 28 11:09:06.297045 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:06.297002 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" podStartSLOduration=2.296989135 podStartE2EDuration="2.296989135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:06.295628141 +0000 UTC m=+2743.428287427" watchObservedRunningTime="2026-07-28 11:09:06.296989135 +0000 UTC m=+2743.429648414" Jul 28 11:09:08.410354 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:08.410311 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:08.907459 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:08.907424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:09.229996 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:09.229966 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:09.431644 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:09.431613 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:12.280913 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.280884 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:12.623071 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.622990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:12.723168 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.723115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:12.773851 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.773723 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:09:12.778214 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.778184 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.781346 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.781316 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-86170f-serving-cert\"" Jul 28 11:09:12.789332 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.789307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:09:12.823086 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.823053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:12.829401 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.829366 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9vf\" (UniqueName: \"kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.829516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.829452 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.829516 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.829480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.930302 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.930238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9vf\" (UniqueName: \"kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.930302 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.930296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.930440 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.930405 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.930551 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:12.930529 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-serving-cert: secret "maas-api-e2e-api-b-86170f-serving-cert" not found Jul 28 11:09:12.930615 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:12.930601 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls podName:a75cbc0e-0eda-4801-ab74-077a1aaebda4 nodeName:}" failed. No retries permitted until 2026-07-28 11:09:13.430586376 +0000 UTC m=+2750.563245634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls") pod "maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" (UID: "a75cbc0e-0eda-4801-ab74-077a1aaebda4") : secret "maas-api-e2e-api-b-86170f-serving-cert" not found Jul 28 11:09:12.930830 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.930814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:12.941341 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:12.941322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9vf\" (UniqueName: \"kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:13.101295 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.101267 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:13.307197 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.307162 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:13.401904 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.401872 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:13.435161 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.435133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:13.437902 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.437870 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") pod \"maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:13.696720 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.696636 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:13.832277 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:13.832251 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:09:13.834658 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:09:13.834620 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75cbc0e_0eda_4801_ab74_077a1aaebda4.slice/crio-001283b861db4d14fd328224a3004abc7cfd592960026209dfdd214e9cca9ffa WatchSource:0}: Error finding container 001283b861db4d14fd328224a3004abc7cfd592960026209dfdd214e9cca9ffa: Status 404 returned error can't find the container with id 001283b861db4d14fd328224a3004abc7cfd592960026209dfdd214e9cca9ffa Jul 28 11:09:14.314951 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:14.314900 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" event={"ID":"a75cbc0e-0eda-4801-ab74-077a1aaebda4","Type":"ContainerStarted","Data":"001283b861db4d14fd328224a3004abc7cfd592960026209dfdd214e9cca9ffa"} Jul 28 11:09:15.322138 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:15.322089 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" event={"ID":"a75cbc0e-0eda-4801-ab74-077a1aaebda4","Type":"ContainerStarted","Data":"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d"} Jul 28 11:09:15.322509 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:15.322159 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:15.338058 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:15.338013 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" podStartSLOduration=2.431793401 podStartE2EDuration="3.337998618s" totalImagesPullingTime="906.205217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:15.337293464 +0000 UTC m=+2752.469952744" watchObservedRunningTime="2026-07-28 11:09:15.337998618 +0000 UTC m=+2752.470657898" Jul 28 11:09:17.172478 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:17.172449 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:17.277173 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:17.277148 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:21.332923 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:21.332885 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:09:40.982628 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:40.982591 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:40.983086 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:40.982924 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" podUID="a552659c-3442-4c03-b4a2-db032241a747" containerName="maas-api" containerID="cri-o://9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892" gracePeriod=30 Jul 28 11:09:41.017670 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.017640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:41.049703 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.049664 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:41.050522 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.049943 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" podUID="512b7d47-8758-4041-9b4d-f234c7f35929" containerName="payload-pre-processing" containerID="cri-o://9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd" gracePeriod=30 Jul 28 11:09:41.118379 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.118347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:41.173105 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.173079 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:41.306179 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.306154 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:41.365364 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.365337 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:41.371015 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.370995 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") pod \"a552659c-3442-4c03-b4a2-db032241a747\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " Jul 28 11:09:41.371109 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.371040 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca\") pod \"a552659c-3442-4c03-b4a2-db032241a747\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " Jul 28 11:09:41.371109 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.371065 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg68w\" (UniqueName: \"kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w\") pod \"a552659c-3442-4c03-b4a2-db032241a747\" (UID: \"a552659c-3442-4c03-b4a2-db032241a747\") " Jul 28 11:09:41.371378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.371351 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a552659c-3442-4c03-b4a2-db032241a747" (UID: "a552659c-3442-4c03-b4a2-db032241a747"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:09:41.373221 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.373198 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a552659c-3442-4c03-b4a2-db032241a747" (UID: "a552659c-3442-4c03-b4a2-db032241a747"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:09:41.373305 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.373285 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w" (OuterVolumeSpecName: "kube-api-access-fg68w") pod "a552659c-3442-4c03-b4a2-db032241a747" (UID: "a552659c-3442-4c03-b4a2-db032241a747"). InnerVolumeSpecName "kube-api-access-fg68w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:09:41.471236 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471211 2567 generic.go:356] "Generic (PLEG): container finished" podID="512b7d47-8758-4041-9b4d-f234c7f35929" containerID="9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd" exitCode=0 Jul 28 11:09:41.471365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" event={"ID":"512b7d47-8758-4041-9b4d-f234c7f35929","Type":"ContainerDied","Data":"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd"} Jul 28 11:09:41.471365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471272 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" Jul 28 11:09:41.471365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw" event={"ID":"512b7d47-8758-4041-9b4d-f234c7f35929","Type":"ContainerDied","Data":"5f3eb6f2ae32791f6d6d29e39dae0d907a1396a1bebb833c1e352a8128164d05"} Jul 28 11:09:41.471365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471317 2567 scope.go:117] "RemoveContainer" containerID="9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd" Jul 28 11:09:41.471575 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471527 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcp9m\" (UniqueName: \"kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m\") pod \"512b7d47-8758-4041-9b4d-f234c7f35929\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " Jul 28 11:09:41.471630 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471613 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume\") pod \"512b7d47-8758-4041-9b4d-f234c7f35929\" (UID: \"512b7d47-8758-4041-9b4d-f234c7f35929\") " Jul 28 11:09:41.471920 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471898 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a552659c-3442-4c03-b4a2-db032241a747-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:09:41.472027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471920 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a552659c-3442-4c03-b4a2-db032241a747-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:09:41.472027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.471935 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg68w\" (UniqueName: \"kubernetes.io/projected/a552659c-3442-4c03-b4a2-db032241a747-kube-api-access-fg68w\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:09:41.472588 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.472458 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "512b7d47-8758-4041-9b4d-f234c7f35929" (UID: "512b7d47-8758-4041-9b4d-f234c7f35929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:09:41.472834 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.472813 2567 generic.go:356] "Generic (PLEG): container finished" podID="a552659c-3442-4c03-b4a2-db032241a747" containerID="9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892" exitCode=0 Jul 28 11:09:41.472937 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.472850 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" Jul 28 11:09:41.472937 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.472876 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" event={"ID":"a552659c-3442-4c03-b4a2-db032241a747","Type":"ContainerDied","Data":"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892"} Jul 28 11:09:41.472937 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.472906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc" event={"ID":"a552659c-3442-4c03-b4a2-db032241a747","Type":"ContainerDied","Data":"65e470cfe04b36dee5e28daea8443c9a772cabb3e9417da701d30a5e99c160ae"} Jul 28 11:09:41.474011 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.473991 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m" (OuterVolumeSpecName: "kube-api-access-wcp9m") pod "512b7d47-8758-4041-9b4d-f234c7f35929" (UID: "512b7d47-8758-4041-9b4d-f234c7f35929"). InnerVolumeSpecName "kube-api-access-wcp9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:09:41.481043 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.481027 2567 scope.go:117] "RemoveContainer" containerID="9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd" Jul 28 11:09:41.481288 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:41.481271 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd\": container with ID starting with 9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd not found: ID does not exist" containerID="9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd" Jul 28 11:09:41.481337 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.481296 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd"} err="failed to get container status \"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd\": rpc error: code = NotFound desc = could not find container \"9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd\": container with ID starting with 9de838370f9f1761ee89cb082a4f117ec57a714b82cfd42dc70fb0b2f195c3cd not found: ID does not exist" Jul 28 11:09:41.481337 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.481312 2567 scope.go:117] "RemoveContainer" containerID="9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892" Jul 28 11:09:41.491913 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.491897 2567 scope.go:117] "RemoveContainer" containerID="9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892" Jul 28 11:09:41.492140 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:09:41.492122 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892\": container with ID starting with 9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892 not found: ID does not exist" containerID="9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892" Jul 28 11:09:41.492188 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.492145 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892"} err="failed to get container status \"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892\": rpc error: code = NotFound desc = could not find container \"9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892\": container with ID starting with 9a7e0190c6683e2ed7c13334a32e3943a017880dfaa6577b055c5c85673a8892 not found: ID does not exist" Jul 28 11:09:41.495916 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.495896 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:41.500132 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.500111 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1912e6-5b8978d957-sttqc"] Jul 28 11:09:41.572590 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.572542 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512b7d47-8758-4041-9b4d-f234c7f35929-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:09:41.572590 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.572562 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcp9m\" (UniqueName: \"kubernetes.io/projected/512b7d47-8758-4041-9b4d-f234c7f35929-kube-api-access-wcp9m\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:09:41.793306 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.793283 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:41.798635 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:41.798612 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1912e6-6d5ccbdd9b-c8nrw"] Jul 28 11:09:43.386510 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:43.386479 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512b7d47-8758-4041-9b4d-f234c7f35929" path="/var/lib/kubelet/pods/512b7d47-8758-4041-9b4d-f234c7f35929/volumes" Jul 28 11:09:43.391765 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:43.387337 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a552659c-3442-4c03-b4a2-db032241a747" path="/var/lib/kubelet/pods/a552659c-3442-4c03-b4a2-db032241a747/volumes" Jul 28 11:09:45.404329 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:45.404245 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:45.829694 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:45.829659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:51.517976 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:51.517937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:51.579167 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:51.579141 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:51.779729 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:51.779642 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:09:51.997529 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:09:51.997499 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:07.528998 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.528962 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:10:07.529495 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.529209 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" podUID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" containerName="maas-api" containerID="cri-o://11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d" gracePeriod=30 Jul 28 11:10:07.559019 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.558986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:07.626851 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.626815 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:07.690034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.689985 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:07.783457 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.783397 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:10:07.898014 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.897984 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") pod \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " Jul 28 11:10:07.898014 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.898017 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc9vf\" (UniqueName: \"kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf\") pod \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " Jul 28 11:10:07.898222 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.898038 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca\") pod \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\" (UID: \"a75cbc0e-0eda-4801-ab74-077a1aaebda4\") " Jul 28 11:10:07.898428 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.898406 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a75cbc0e-0eda-4801-ab74-077a1aaebda4" (UID: "a75cbc0e-0eda-4801-ab74-077a1aaebda4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:10:07.900523 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.900495 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a75cbc0e-0eda-4801-ab74-077a1aaebda4" (UID: "a75cbc0e-0eda-4801-ab74-077a1aaebda4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:10:07.900616 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.900517 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf" (OuterVolumeSpecName: "kube-api-access-wc9vf") pod "a75cbc0e-0eda-4801-ab74-077a1aaebda4" (UID: "a75cbc0e-0eda-4801-ab74-077a1aaebda4"). InnerVolumeSpecName "kube-api-access-wc9vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:10:07.998791 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.998765 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75cbc0e-0eda-4801-ab74-077a1aaebda4-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:10:07.998791 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.998790 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9vf\" (UniqueName: \"kubernetes.io/projected/a75cbc0e-0eda-4801-ab74-077a1aaebda4-kube-api-access-wc9vf\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:10:07.998922 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:07.998806 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75cbc0e-0eda-4801-ab74-077a1aaebda4-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:10:08.618926 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.618885 2567 generic.go:356] "Generic (PLEG): container finished" podID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" containerID="11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d" exitCode=0 Jul 28 11:10:08.619309 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.618951 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" Jul 28 11:10:08.619309 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.618969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" event={"ID":"a75cbc0e-0eda-4801-ab74-077a1aaebda4","Type":"ContainerDied","Data":"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d"} Jul 28 11:10:08.619309 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.619024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz" event={"ID":"a75cbc0e-0eda-4801-ab74-077a1aaebda4","Type":"ContainerDied","Data":"001283b861db4d14fd328224a3004abc7cfd592960026209dfdd214e9cca9ffa"} Jul 28 11:10:08.619309 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.619045 2567 scope.go:117] "RemoveContainer" containerID="11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d" Jul 28 11:10:08.628972 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.628954 2567 scope.go:117] "RemoveContainer" containerID="11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d" Jul 28 11:10:08.629250 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:10:08.629223 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d\": container with ID starting with 11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d not found: ID does not exist" containerID="11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d" Jul 28 11:10:08.629308 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.629260 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d"} err="failed to get container status \"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d\": rpc error: code = NotFound desc = could not find container \"11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d\": container with ID starting with 11ca63d549ccf4cfdfaa017db9e234cb3d9fc7700f7f2a3510a69bad7132c72d not found: ID does not exist" Jul 28 11:10:08.642620 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.642551 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:10:08.648153 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:08.648130 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86170f-7ccf4f8c68-sdrcz"] Jul 28 11:10:09.387416 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:09.387383 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" path="/var/lib/kubelet/pods/a75cbc0e-0eda-4801-ab74-077a1aaebda4/volumes" Jul 28 11:10:12.200361 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:12.200326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:12.610704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:12.610672 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:18.454708 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:18.454662 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:18.555414 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:18.555382 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:18.741020 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:18.740990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:18.994452 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:18.994372 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:19.367365 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:19.367282 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:19.695030 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:19.694941 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:19.912878 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:19.912845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:22.235477 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.235415 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:22.321246 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.321208 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:22.327442 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.327411 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:10:22.327998 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.327975 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" containerName="maas-api" Jul 28 11:10:22.327998 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.327999 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" containerName="maas-api" Jul 28 11:10:22.328123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328025 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512b7d47-8758-4041-9b4d-f234c7f35929" containerName="payload-pre-processing" Jul 28 11:10:22.328123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328037 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="512b7d47-8758-4041-9b4d-f234c7f35929" containerName="payload-pre-processing" Jul 28 11:10:22.328123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328069 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a552659c-3442-4c03-b4a2-db032241a747" containerName="maas-api" Jul 28 11:10:22.328123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328077 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a552659c-3442-4c03-b4a2-db032241a747" containerName="maas-api" Jul 28 11:10:22.328249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328163 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="512b7d47-8758-4041-9b4d-f234c7f35929" containerName="payload-pre-processing" Jul 28 11:10:22.328249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328174 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a552659c-3442-4c03-b4a2-db032241a747" containerName="maas-api" Jul 28 11:10:22.328249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.328185 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75cbc0e-0eda-4801-ab74-077a1aaebda4" containerName="maas-api" Jul 28 11:10:22.332828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.332689 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.335580 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.335561 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4cda06-serving-cert\"" Jul 28 11:10:22.343993 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.343975 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:10:22.399029 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.399005 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:22.403161 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.403135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.403278 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.403170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmgr\" (UniqueName: \"kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.403334 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.403291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.504496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.504407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.504496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.504472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.504682 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.504507 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmgr\" (UniqueName: \"kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.505112 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.505089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.506952 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.506936 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.513312 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.513282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmgr\" (UniqueName: \"kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr\") pod \"maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.547089 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.547065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:22.644482 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.644459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:22.795419 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.795391 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:10:22.797095 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:10:22.797070 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e8a39f_2b4a_4017_b6fb_b83061762610.slice/crio-7a73031392a803ae67897f177699c466f8b7f0eb47cf23b5358763a83889ff92 WatchSource:0}: Error finding container 7a73031392a803ae67897f177699c466f8b7f0eb47cf23b5358763a83889ff92: Status 404 returned error can't find the container with id 7a73031392a803ae67897f177699c466f8b7f0eb47cf23b5358763a83889ff92 Jul 28 11:10:22.993159 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:22.993124 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:23.085703 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.085613 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:10:23.087285 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.087259 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:23.090478 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.090451 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.093019 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.093001 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4cda06\"" Jul 28 11:10:23.100830 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.100796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:10:23.111274 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.111245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.111391 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.111323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtgc\" (UniqueName: \"kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.211878 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.211853 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.212001 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.211915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtgc\" (UniqueName: \"kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.214554 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.214524 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4cda06\"" Jul 28 11:10:23.220182 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.220156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtgc\" (UniqueName: \"kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.223357 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.223332 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.402512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.402437 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.546653 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.546619 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:10:23.546815 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:10:23.546758 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55a1e8c_bce5_4462_a3ac_f042e65ae232.slice/crio-a4b6d416046e2158eabe242aef26bf470f27ff0ae35a1d0d2eeec94c9eda0dda WatchSource:0}: Error finding container a4b6d416046e2158eabe242aef26bf470f27ff0ae35a1d0d2eeec94c9eda0dda: Status 404 returned error can't find the container with id a4b6d416046e2158eabe242aef26bf470f27ff0ae35a1d0d2eeec94c9eda0dda Jul 28 11:10:23.703625 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.703543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" event={"ID":"a55a1e8c-bce5-4462-a3ac-f042e65ae232","Type":"ContainerStarted","Data":"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5"} Jul 28 11:10:23.703625 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.703613 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:23.703842 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.703640 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" event={"ID":"a55a1e8c-bce5-4462-a3ac-f042e65ae232","Type":"ContainerStarted","Data":"a4b6d416046e2158eabe242aef26bf470f27ff0ae35a1d0d2eeec94c9eda0dda"} Jul 28 11:10:23.705052 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.705030 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" event={"ID":"d5e8a39f-2b4a-4017-b6fb-b83061762610","Type":"ContainerStarted","Data":"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751"} Jul 28 11:10:23.705130 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.705068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" event={"ID":"d5e8a39f-2b4a-4017-b6fb-b83061762610","Type":"ContainerStarted","Data":"7a73031392a803ae67897f177699c466f8b7f0eb47cf23b5358763a83889ff92"} Jul 28 11:10:23.705180 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.705160 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:23.721458 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.721412 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" podStartSLOduration=0.721396573 podStartE2EDuration="721.396573ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:23.71902806 +0000 UTC m=+2820.851687331" watchObservedRunningTime="2026-07-28 11:10:23.721396573 +0000 UTC m=+2820.854055830" Jul 28 11:10:23.735753 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:23.735691 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" podStartSLOduration=1.3947683 podStartE2EDuration="1.735678486s" totalImagesPullingTime="340.910186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:23.734538919 +0000 UTC m=+2820.867198211" watchObservedRunningTime="2026-07-28 11:10:23.735678486 +0000 UTC m=+2820.868337825" Jul 28 11:10:26.765926 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:26.765880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:27.112429 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:27.112342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:27.436331 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:27.436221 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:28.041704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:28.041673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:29.717193 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:29.717165 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:10:30.035324 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.035250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:30.135374 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.135344 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:30.167815 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.167772 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:10:30.172153 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.172131 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.174703 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.174686 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-683bdc-serving-cert\"" Jul 28 11:10:30.181012 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.180887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:10:30.237403 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.237373 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:30.268798 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.268770 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.268928 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.268833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.268928 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.268915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzll\" (UniqueName: \"kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.369938 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.369859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.369938 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.369925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.370132 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.369958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzll\" (UniqueName: \"kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.370132 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:10:30.370035 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-serving-cert: secret "maas-api-e2e-shared-b-683bdc-serving-cert" not found Jul 28 11:10:30.370132 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:10:30.370109 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls podName:380b5377-52fe-4f19-9266-aad48685dce0 nodeName:}" failed. No retries permitted until 2026-07-28 11:10:30.870087037 +0000 UTC m=+2828.002746294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls") pod "maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" (UID: "380b5377-52fe-4f19-9266-aad48685dce0") : secret "maas-api-e2e-shared-b-683bdc-serving-cert" not found Jul 28 11:10:30.370441 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.370421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.378316 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.378295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzll\" (UniqueName: \"kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.470518 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.470488 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:30.796799 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.796768 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:30.873972 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.873944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.876464 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.876438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") pod \"maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:30.920075 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:30.920046 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:31.086913 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.086828 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:31.228161 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.228137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:10:31.229886 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:10:31.229850 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380b5377_52fe_4f19_9266_aad48685dce0.slice/crio-ab1fc2701d025f5eb93e09cd779f4c8be5e040aeb36b1fed9d20d0cede800c21 WatchSource:0}: Error finding container ab1fc2701d025f5eb93e09cd779f4c8be5e040aeb36b1fed9d20d0cede800c21: Status 404 returned error can't find the container with id ab1fc2701d025f5eb93e09cd779f4c8be5e040aeb36b1fed9d20d0cede800c21 Jul 28 11:10:31.752584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.752552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" event={"ID":"380b5377-52fe-4f19-9266-aad48685dce0","Type":"ContainerStarted","Data":"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5"} Jul 28 11:10:31.752759 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.752613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" event={"ID":"380b5377-52fe-4f19-9266-aad48685dce0","Type":"ContainerStarted","Data":"ab1fc2701d025f5eb93e09cd779f4c8be5e040aeb36b1fed9d20d0cede800c21"} Jul 28 11:10:31.752759 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.752658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:31.772179 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:31.772134 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" podStartSLOduration=1.4291781000000001 podStartE2EDuration="1.772119679s" totalImagesPullingTime="342.941579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:31.770068944 +0000 UTC m=+2828.902728224" watchObservedRunningTime="2026-07-28 11:10:31.772119679 +0000 UTC m=+2828.904778958" Jul 28 11:10:34.574296 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:34.574250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:10:34.713906 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:34.713804 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:10:37.763441 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:37.763409 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:10:50.978854 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:10:50.978819 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:11:10.477664 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:11:10.477625 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:11:10.578882 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:11:10.578825 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:11:16.677126 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:11:16.677092 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:11:47.750305 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:11:47.750258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:13:39.820689 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:13:39.820656 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:15:21.683990 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:15:21.683955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:15:27.247541 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:15:27.247499 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:15:28.241020 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:15:28.240979 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:15:45.266378 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:15:45.266283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:16:17.261898 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:16:17.261862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:14.924148 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:14.924046 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:28.616608 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:28.616549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:29.551685 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:29.551654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:30.445700 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:30.445669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:44.982615 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:44.982578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:49.455101 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:49.455070 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:17:49.658580 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:17:49.658539 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:18:25.956316 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:18:25.956283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:31.218306 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:31.218272 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:32.099697 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:32.099663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:52.946258 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:52.946222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:53.308935 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:53.308903 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:53.591790 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:53.591689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.290374 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.290331 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.373141 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.373114 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:19:56.374644 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.374619 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.377037 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.377017 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.379712 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.379695 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1c9694-serving-cert\"" Jul 28 11:19:56.386838 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.386816 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:19:56.451160 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.451131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.523526 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.523495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7pc\" (UniqueName: \"kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.523670 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.523530 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.523670 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.523564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.624413 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.624339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7pc\" (UniqueName: \"kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.624413 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.624375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.624413 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.624398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.624643 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:19:56.624499 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-serving-cert: secret "maas-api-e2e-ipp-a-1c9694-serving-cert" not found Jul 28 11:19:56.624643 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:19:56.624556 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls podName:8d4025dc-a75b-4a8b-af45-bc374d294e37 nodeName:}" failed. No retries permitted until 2026-07-28 11:19:57.12454074 +0000 UTC m=+3394.257199997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls") pod "maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" (UID: "8d4025dc-a75b-4a8b-af45-bc374d294e37") : secret "maas-api-e2e-ipp-a-1c9694-serving-cert" not found Jul 28 11:19:56.625034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.625016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.635369 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.635351 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7pc\" (UniqueName: \"kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:56.863767 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.863704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.961048 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.961014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:56.963192 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.963168 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:19:56.966730 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.966714 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:56.969720 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.969696 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1c9694\"" Jul 28 11:19:56.969844 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.969802 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1c9694-dockercfg-crl7d\"" Jul 28 11:19:56.979427 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.979404 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:19:56.999835 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:56.999812 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:19:57.003192 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.003176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.014248 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.014228 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:19:57.054871 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.054420 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:19:57.129586 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.129556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ktl\" (UniqueName: \"kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.129752 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.129608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.129752 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.129720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.129837 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.129768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r5s\" (UniqueName: \"kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.129873 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.129838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:57.132828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.132790 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:57.231208 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231138 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.231208 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r5s\" (UniqueName: \"kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.231356 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ktl\" (UniqueName: \"kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.231356 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.231975 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.232073 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.231978 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.239234 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.239209 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r5s\" (UniqueName: \"kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s\") pod \"payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.239338 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.239317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ktl\" (UniqueName: \"kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl\") pod \"payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.278110 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.278091 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.290182 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.290158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:19:57.314256 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.314230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.440064 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.439928 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:19:57.445795 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:19:57.445703 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5775b75d_2572_4ae4_9aab_f7b015a348b0.slice/crio-ba6e99959717520ef901b7f89c69d246a3c9b3fa777c931bed4e0f6114196d54 WatchSource:0}: Error finding container ba6e99959717520ef901b7f89c69d246a3c9b3fa777c931bed4e0f6114196d54: Status 404 returned error can't find the container with id ba6e99959717520ef901b7f89c69d246a3c9b3fa777c931bed4e0f6114196d54 Jul 28 11:19:57.473960 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.473592 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:19:57.474311 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:19:57.474269 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4025dc_a75b_4a8b_af45_bc374d294e37.slice/crio-b6826f09bbbb705922b3510d053efccbb1819331ace78445745977b68c07f364 WatchSource:0}: Error finding container b6826f09bbbb705922b3510d053efccbb1819331ace78445745977b68c07f364: Status 404 returned error can't find the container with id b6826f09bbbb705922b3510d053efccbb1819331ace78445745977b68c07f364 Jul 28 11:19:57.475777 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.475758 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 11:19:57.503995 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.503967 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:19:57.505244 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:19:57.505211 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ded5521_3f39_4c42_ba24_2482687236f6.slice/crio-83a4607680aac3fa2d23fcbc19a10dcb36356119034710575772de0ab1399038 WatchSource:0}: Error finding container 83a4607680aac3fa2d23fcbc19a10dcb36356119034710575772de0ab1399038: Status 404 returned error can't find the container with id 83a4607680aac3fa2d23fcbc19a10dcb36356119034710575772de0ab1399038 Jul 28 11:19:57.794264 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.794170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" event={"ID":"8ded5521-3f39-4c42-ba24-2482687236f6","Type":"ContainerStarted","Data":"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0"} Jul 28 11:19:57.794264 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.794236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" event={"ID":"8ded5521-3f39-4c42-ba24-2482687236f6","Type":"ContainerStarted","Data":"83a4607680aac3fa2d23fcbc19a10dcb36356119034710575772de0ab1399038"} Jul 28 11:19:57.794488 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.794270 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:19:57.795582 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.795554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" event={"ID":"8d4025dc-a75b-4a8b-af45-bc374d294e37","Type":"ContainerStarted","Data":"b6826f09bbbb705922b3510d053efccbb1819331ace78445745977b68c07f364"} Jul 28 11:19:57.797122 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.797064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" event={"ID":"5775b75d-2572-4ae4-9aab-f7b015a348b0","Type":"ContainerStarted","Data":"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6"} Jul 28 11:19:57.797122 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.797109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" event={"ID":"5775b75d-2572-4ae4-9aab-f7b015a348b0","Type":"ContainerStarted","Data":"ba6e99959717520ef901b7f89c69d246a3c9b3fa777c931bed4e0f6114196d54"} Jul 28 11:19:57.797278 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.797144 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:19:57.811038 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.810997 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" podStartSLOduration=1.810979873 podStartE2EDuration="1.810979873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:19:57.809489873 +0000 UTC m=+3394.942149153" watchObservedRunningTime="2026-07-28 11:19:57.810979873 +0000 UTC m=+3394.943639143" Jul 28 11:19:57.826412 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:57.826360 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" podStartSLOduration=1.826346745 podStartE2EDuration="1.826346745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:19:57.824550702 +0000 UTC m=+3394.957209982" watchObservedRunningTime="2026-07-28 11:19:57.826346745 +0000 UTC m=+3394.959006023" Jul 28 11:19:58.804903 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:58.804866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" event={"ID":"8d4025dc-a75b-4a8b-af45-bc374d294e37","Type":"ContainerStarted","Data":"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370"} Jul 28 11:19:58.823267 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:58.823218 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" podStartSLOduration=2.363171364 podStartE2EDuration="2.823201934s" totalImagesPullingTime="460.03057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:19:58.822404852 +0000 UTC m=+3395.955064145" watchObservedRunningTime="2026-07-28 11:19:58.823201934 +0000 UTC m=+3395.955861213" Jul 28 11:19:59.811642 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:19:59.811610 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:20:00.895910 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:00.895825 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:05.822521 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:05.822487 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:20:08.806789 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:08.806728 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:20:08.807199 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:08.806919 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:20:11.795214 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:11.795175 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:12.098360 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:12.098285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:12.198717 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:12.198690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.141959 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.141855 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.218843 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.218806 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.220794 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.220767 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:20:15.224416 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.224398 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.226993 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.226975 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a93417-serving-cert\"" Jul 28 11:20:15.238037 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.238019 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:20:15.283625 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.283595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4t6r\" (UniqueName: \"kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.283762 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.283648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.283762 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.283670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.293884 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.293858 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.384129 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.384104 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4t6r\" (UniqueName: \"kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.384249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.384157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.384249 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.384178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.384332 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:20:15.384311 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-serving-cert: secret "maas-api-e2e-ipp-b-a93417-serving-cert" not found Jul 28 11:20:15.384380 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:20:15.384361 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls podName:9fd95959-4371-4e9e-9aaf-a675d2a341f8 nodeName:}" failed. No retries permitted until 2026-07-28 11:20:15.88434691 +0000 UTC m=+3413.017006167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls") pod "maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" (UID: "9fd95959-4371-4e9e-9aaf-a675d2a341f8") : secret "maas-api-e2e-ipp-b-a93417-serving-cert" not found Jul 28 11:20:15.384828 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.384809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.393925 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.393876 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4t6r\" (UniqueName: \"kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.523497 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.523467 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.819540 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.819508 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.889476 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.889442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.892094 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.892066 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:15.915124 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.915097 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:15.920647 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.920624 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:20:15.924515 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.924495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:15.927353 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.927332 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a93417\"" Jul 28 11:20:15.927441 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.927421 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a93417-dockercfg-7drf4\"" Jul 28 11:20:15.937314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.937294 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:20:15.990488 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.990460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5bk\" (UniqueName: \"kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:15.990604 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:15.990501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.091584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.091499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5bk\" (UniqueName: \"kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.091584 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.091567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.092314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.092293 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.103796 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.103776 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5bk\" (UniqueName: \"kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk\") pod \"payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.134896 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.134871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:16.236107 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.236074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.280213 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.280184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:20:16.280369 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:20:16.280346 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd95959_4371_4e9e_9aaf_a675d2a341f8.slice/crio-18c8e0706802e4d990d6a88097335c50c93639d2c7ec2599f45d64e42b79a6ea WatchSource:0}: Error finding container 18c8e0706802e4d990d6a88097335c50c93639d2c7ec2599f45d64e42b79a6ea: Status 404 returned error can't find the container with id 18c8e0706802e4d990d6a88097335c50c93639d2c7ec2599f45d64e42b79a6ea Jul 28 11:20:16.377030 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.377005 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:20:16.379475 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:20:16.379451 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3ab7f4_e571_48fd_bc34_acd625e4910c.slice/crio-1992ca91979203a9f8197bd4e85c93af89b02e1b9a831159c1b8df7753d9cb97 WatchSource:0}: Error finding container 1992ca91979203a9f8197bd4e85c93af89b02e1b9a831159c1b8df7753d9cb97: Status 404 returned error can't find the container with id 1992ca91979203a9f8197bd4e85c93af89b02e1b9a831159c1b8df7753d9cb97 Jul 28 11:20:16.911381 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.911339 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" event={"ID":"ab3ab7f4-e571-48fd-bc34-acd625e4910c","Type":"ContainerStarted","Data":"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941"} Jul 28 11:20:16.911549 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.911406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" event={"ID":"ab3ab7f4-e571-48fd-bc34-acd625e4910c","Type":"ContainerStarted","Data":"1992ca91979203a9f8197bd4e85c93af89b02e1b9a831159c1b8df7753d9cb97"} Jul 28 11:20:16.911549 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.911463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:16.913431 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.913408 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" event={"ID":"9fd95959-4371-4e9e-9aaf-a675d2a341f8","Type":"ContainerStarted","Data":"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd"} Jul 28 11:20:16.913602 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.913448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" event={"ID":"9fd95959-4371-4e9e-9aaf-a675d2a341f8","Type":"ContainerStarted","Data":"18c8e0706802e4d990d6a88097335c50c93639d2c7ec2599f45d64e42b79a6ea"} Jul 28 11:20:16.913602 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.913476 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:16.931276 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.931205 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" podStartSLOduration=1.931186925 podStartE2EDuration="1.931186925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:20:16.927707121 +0000 UTC m=+3414.060366412" watchObservedRunningTime="2026-07-28 11:20:16.931186925 +0000 UTC m=+3414.063846204" Jul 28 11:20:16.947104 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:16.947055 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" podStartSLOduration=1.530410445 podStartE2EDuration="1.947041251s" totalImagesPullingTime="416.630806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:20:16.945010182 +0000 UTC m=+3414.077669464" watchObservedRunningTime="2026-07-28 11:20:16.947041251 +0000 UTC m=+3414.079700559" Jul 28 11:20:19.630036 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:19.629997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:22.924090 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:22.924060 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:20:27.921057 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:27.921018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:20:37.330534 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:37.330504 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log" Jul 28 11:20:41.841347 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:41.841313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:20:48.868184 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:20:48.868152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:34.971990 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:34.971959 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2_8ded5521-3f39-4c42-ba24-2482687236f6/payload-processing/0.log" Jul 28 11:21:35.110433 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:35.110402 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log" Jul 28 11:21:35.567818 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:35.567779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:35.945225 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:35.945138 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:36.911655 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:36.911613 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:38.659480 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:38.659442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:38.755787 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:38.755727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:40.412409 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.412380 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:21:40.416114 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.416094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.419872 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.419853 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-0a6eb5-serving-cert\"" Jul 28 11:21:40.430664 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.430642 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:21:40.453125 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.453101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:40.549077 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.549048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2bd\" (UniqueName: \"kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.549216 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.549091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.549361 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.549336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.650728 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.650701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.650857 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.650793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2bd\" (UniqueName: \"kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.650857 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.650815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.651425 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.651403 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.653402 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.653381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.665961 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.665908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2bd\" (UniqueName: \"kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd\") pod \"maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.726726 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.726703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:40.741105 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.741074 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:40.870704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.870438 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:21:40.875312 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:21:40.875285 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea75d814_724b_4d65_9adb_ce718849eda7.slice/crio-cfb47eed914d53df791f29f372067be3b354ecd1c710d91893f4e7abf615d232 WatchSource:0}: Error finding container cfb47eed914d53df791f29f372067be3b354ecd1c710d91893f4e7abf615d232: Status 404 returned error can't find the container with id cfb47eed914d53df791f29f372067be3b354ecd1c710d91893f4e7abf615d232 Jul 28 11:21:40.960832 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:40.960790 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:41.055664 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.055634 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:21:41.056166 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.056131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:41.060528 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.060512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.063388 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.063367 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-0a6eb5\"" Jul 28 11:21:41.063509 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.063490 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-0a6eb5-dockercfg-g4pmv\"" Jul 28 11:21:41.072382 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.072360 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:21:41.087906 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.087882 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:21:41.093844 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.092222 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.108492 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.108463 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:21:41.157318 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.157282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.157510 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.157331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86s9q\" (UniqueName: \"kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.157510 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.157375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.158262 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.158001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgb42\" (UniqueName: \"kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.258649 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.258622 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.258813 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.258725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgb42\" (UniqueName: \"kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.258867 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.258845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.258924 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.258874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86s9q\" (UniqueName: \"kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.259432 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.259409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.259582 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.259562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.268441 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.268422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgb42\" (UniqueName: \"kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42\") pod \"payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.268657 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.268636 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86s9q\" (UniqueName: \"kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q\") pod \"payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.373844 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.373812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:41.389464 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.389429 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:41.389587 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.389512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" event={"ID":"ea75d814-724b-4d65-9adb-ce718849eda7","Type":"ContainerStarted","Data":"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075"} Jul 28 11:21:41.389587 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.389549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" event={"ID":"ea75d814-724b-4d65-9adb-ce718849eda7","Type":"ContainerStarted","Data":"cfb47eed914d53df791f29f372067be3b354ecd1c710d91893f4e7abf615d232"} Jul 28 11:21:41.406093 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.406048 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" podStartSLOduration=1.051398876 podStartE2EDuration="1.406032638s" totalImagesPullingTime="354.633762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:21:41.405140241 +0000 UTC m=+3498.537799521" watchObservedRunningTime="2026-07-28 11:21:41.406032638 +0000 UTC m=+3498.538691916" Jul 28 11:21:41.408264 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.408245 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:41.565832 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.565802 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:21:41.567891 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:21:41.567838 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b1345d_e571_4310_aba4_a8bb70a8ab33.slice/crio-dc3be5538cde5efa0caa8837eda32b2d776007b882b45de67620ff42f31d7658 WatchSource:0}: Error finding container dc3be5538cde5efa0caa8837eda32b2d776007b882b45de67620ff42f31d7658: Status 404 returned error can't find the container with id dc3be5538cde5efa0caa8837eda32b2d776007b882b45de67620ff42f31d7658 Jul 28 11:21:41.795849 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:41.795819 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:21:41.796669 ip-10-0-130-55 kubenswrapper[2567]: W0728 11:21:41.796637 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06587de_95e0_4668_923d_ec6141f9e91e.slice/crio-4205e87469657ae8bcf370867b499aa3759b69f8931e7c0e3f56215901f285ab WatchSource:0}: Error finding container 4205e87469657ae8bcf370867b499aa3759b69f8931e7c0e3f56215901f285ab: Status 404 returned error can't find the container with id 4205e87469657ae8bcf370867b499aa3759b69f8931e7c0e3f56215901f285ab Jul 28 11:21:42.392112 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.392074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" event={"ID":"d06587de-95e0-4668-923d-ec6141f9e91e","Type":"ContainerStarted","Data":"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175"} Jul 28 11:21:42.392314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.392136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" event={"ID":"d06587de-95e0-4668-923d-ec6141f9e91e","Type":"ContainerStarted","Data":"4205e87469657ae8bcf370867b499aa3759b69f8931e7c0e3f56215901f285ab"} Jul 28 11:21:42.392314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.392162 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:21:42.393660 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.393624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" event={"ID":"69b1345d-e571-4310-aba4-a8bb70a8ab33","Type":"ContainerStarted","Data":"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee"} Jul 28 11:21:42.393822 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.393681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" event={"ID":"69b1345d-e571-4310-aba4-a8bb70a8ab33","Type":"ContainerStarted","Data":"dc3be5538cde5efa0caa8837eda32b2d776007b882b45de67620ff42f31d7658"} Jul 28 11:21:42.393822 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.393808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:42.412889 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.412849 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" podStartSLOduration=1.412835917 podStartE2EDuration="1.412835917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:21:42.410221557 +0000 UTC m=+3499.542880837" watchObservedRunningTime="2026-07-28 11:21:42.412835917 +0000 UTC m=+3499.545495244" Jul 28 11:21:42.431796 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:42.431730 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" podStartSLOduration=1.431711195 podStartE2EDuration="1.431711195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:21:42.427712126 +0000 UTC m=+3499.560371417" watchObservedRunningTime="2026-07-28 11:21:42.431711195 +0000 UTC m=+3499.564370480" Jul 28 11:21:43.167824 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:43.167763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:21:47.399366 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:47.399281 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:21:53.402859 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:53.402831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:21:53.403305 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:21:53.402889 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:22:07.471384 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.471345 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:22:07.471793 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.471532 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" podUID="ea75d814-724b-4d65-9adb-ce718849eda7" containerName="maas-api" containerID="cri-o://f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075" gracePeriod=30 Jul 28 11:22:07.514992 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.514962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:07.562481 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.562448 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:22:07.562769 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.562694 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" podUID="d06587de-95e0-4668-923d-ec6141f9e91e" containerName="payload-processing" containerID="cri-o://c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175" gracePeriod=30 Jul 28 11:22:07.577955 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.577929 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:22:07.578116 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.578098 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" podUID="69b1345d-e571-4310-aba4-a8bb70a8ab33" containerName="payload-pre-processing" containerID="cri-o://240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee" gracePeriod=30 Jul 28 11:22:07.634373 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.634339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:07.737409 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.737251 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:07.851110 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.851035 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:22:07.870173 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.870148 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:22:07.878388 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:07.878368 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:22:08.003129 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003051 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume\") pod \"d06587de-95e0-4668-923d-ec6141f9e91e\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " Jul 28 11:22:08.003129 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003097 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume\") pod \"69b1345d-e571-4310-aba4-a8bb70a8ab33\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " Jul 28 11:22:08.003332 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003146 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86s9q\" (UniqueName: \"kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q\") pod \"69b1345d-e571-4310-aba4-a8bb70a8ab33\" (UID: \"69b1345d-e571-4310-aba4-a8bb70a8ab33\") " Jul 28 11:22:08.003332 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003200 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th2bd\" (UniqueName: \"kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd\") pod \"ea75d814-724b-4d65-9adb-ce718849eda7\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " Jul 28 11:22:08.003332 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003270 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca\") pod \"ea75d814-724b-4d65-9adb-ce718849eda7\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " Jul 28 11:22:08.003332 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003298 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls\") pod \"ea75d814-724b-4d65-9adb-ce718849eda7\" (UID: \"ea75d814-724b-4d65-9adb-ce718849eda7\") " Jul 28 11:22:08.003520 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003347 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgb42\" (UniqueName: \"kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42\") pod \"d06587de-95e0-4668-923d-ec6141f9e91e\" (UID: \"d06587de-95e0-4668-923d-ec6141f9e91e\") " Jul 28 11:22:08.003520 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003467 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d06587de-95e0-4668-923d-ec6141f9e91e" (UID: "d06587de-95e0-4668-923d-ec6141f9e91e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:08.003712 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003651 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69b1345d-e571-4310-aba4-a8bb70a8ab33" (UID: "69b1345d-e571-4310-aba4-a8bb70a8ab33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:08.003712 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.003659 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea75d814-724b-4d65-9adb-ce718849eda7" (UID: "ea75d814-724b-4d65-9adb-ce718849eda7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:08.004142 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.004017 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75d814-724b-4d65-9adb-ce718849eda7-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.004142 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.004041 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06587de-95e0-4668-923d-ec6141f9e91e-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.004142 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.004057 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b1345d-e571-4310-aba4-a8bb70a8ab33-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.005485 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.005460 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q" (OuterVolumeSpecName: "kube-api-access-86s9q") pod "69b1345d-e571-4310-aba4-a8bb70a8ab33" (UID: "69b1345d-e571-4310-aba4-a8bb70a8ab33"). InnerVolumeSpecName "kube-api-access-86s9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:08.005572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.005513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd" (OuterVolumeSpecName: "kube-api-access-th2bd") pod "ea75d814-724b-4d65-9adb-ce718849eda7" (UID: "ea75d814-724b-4d65-9adb-ce718849eda7"). InnerVolumeSpecName "kube-api-access-th2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:08.005572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.005507 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea75d814-724b-4d65-9adb-ce718849eda7" (UID: "ea75d814-724b-4d65-9adb-ce718849eda7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:22:08.005572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.005564 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42" (OuterVolumeSpecName: "kube-api-access-dgb42") pod "d06587de-95e0-4668-923d-ec6141f9e91e" (UID: "d06587de-95e0-4668-923d-ec6141f9e91e"). InnerVolumeSpecName "kube-api-access-dgb42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:08.104627 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.104600 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2bd\" (UniqueName: \"kubernetes.io/projected/ea75d814-724b-4d65-9adb-ce718849eda7-kube-api-access-th2bd\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.104627 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.104624 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea75d814-724b-4d65-9adb-ce718849eda7-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.104787 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.104633 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgb42\" (UniqueName: \"kubernetes.io/projected/d06587de-95e0-4668-923d-ec6141f9e91e-kube-api-access-dgb42\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.104787 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.104644 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86s9q\" (UniqueName: \"kubernetes.io/projected/69b1345d-e571-4310-aba4-a8bb70a8ab33-kube-api-access-86s9q\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:08.552641 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.552611 2567 generic.go:356] "Generic (PLEG): container finished" podID="69b1345d-e571-4310-aba4-a8bb70a8ab33" containerID="240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee" exitCode=0 Jul 28 11:22:08.553027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.552658 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" Jul 28 11:22:08.553027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.552697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" event={"ID":"69b1345d-e571-4310-aba4-a8bb70a8ab33","Type":"ContainerDied","Data":"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee"} Jul 28 11:22:08.553027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.552775 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8" event={"ID":"69b1345d-e571-4310-aba4-a8bb70a8ab33","Type":"ContainerDied","Data":"dc3be5538cde5efa0caa8837eda32b2d776007b882b45de67620ff42f31d7658"} Jul 28 11:22:08.553027 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.552805 2567 scope.go:117] "RemoveContainer" containerID="240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee" Jul 28 11:22:08.554185 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.554166 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea75d814-724b-4d65-9adb-ce718849eda7" containerID="f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075" exitCode=0 Jul 28 11:22:08.554280 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.554209 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" Jul 28 11:22:08.554280 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.554245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" event={"ID":"ea75d814-724b-4d65-9adb-ce718849eda7","Type":"ContainerDied","Data":"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075"} Jul 28 11:22:08.554374 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.554291 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67" event={"ID":"ea75d814-724b-4d65-9adb-ce718849eda7","Type":"ContainerDied","Data":"cfb47eed914d53df791f29f372067be3b354ecd1c710d91893f4e7abf615d232"} Jul 28 11:22:08.555851 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.555826 2567 generic.go:356] "Generic (PLEG): container finished" podID="d06587de-95e0-4668-923d-ec6141f9e91e" containerID="c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175" exitCode=0 Jul 28 11:22:08.555952 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.555858 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" Jul 28 11:22:08.555952 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.555903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" event={"ID":"d06587de-95e0-4668-923d-ec6141f9e91e","Type":"ContainerDied","Data":"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175"} Jul 28 11:22:08.555952 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.555942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s" event={"ID":"d06587de-95e0-4668-923d-ec6141f9e91e","Type":"ContainerDied","Data":"4205e87469657ae8bcf370867b499aa3759b69f8931e7c0e3f56215901f285ab"} Jul 28 11:22:08.569416 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.569356 2567 scope.go:117] "RemoveContainer" containerID="240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee" Jul 28 11:22:08.570075 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:08.569805 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee\": container with ID starting with 240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee not found: ID does not exist" containerID="240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee" Jul 28 11:22:08.570075 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.569840 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee"} err="failed to get container status \"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee\": rpc error: code = NotFound desc = could not find container \"240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee\": container with ID starting with 240684e6c6896c8f08ee4d60062f01abd5d00776b7d8dc386cfd09bcfde820ee not found: ID does not exist" Jul 28 11:22:08.570075 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.569938 2567 scope.go:117] "RemoveContainer" containerID="f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075" Jul 28 11:22:08.579357 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.579330 2567 scope.go:117] "RemoveContainer" containerID="f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075" Jul 28 11:22:08.579590 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:08.579571 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075\": container with ID starting with f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075 not found: ID does not exist" containerID="f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075" Jul 28 11:22:08.579668 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.579600 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075"} err="failed to get container status \"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075\": rpc error: code = NotFound desc = could not find container \"f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075\": container with ID starting with f1e75f84d67fdb7503c0b5c7db5fffca37ec5683779442dc7454ea3dd8c45075 not found: ID does not exist" Jul 28 11:22:08.579668 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.579622 2567 scope.go:117] "RemoveContainer" containerID="c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175" Jul 28 11:22:08.587169 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.587152 2567 scope.go:117] "RemoveContainer" containerID="c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175" Jul 28 11:22:08.587413 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:08.587393 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175\": container with ID starting with c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175 not found: ID does not exist" containerID="c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175" Jul 28 11:22:08.587489 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.587420 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175"} err="failed to get container status \"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175\": rpc error: code = NotFound desc = could not find container \"c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175\": container with ID starting with c20ba5fef5bb571a95d2debaf9004f2eea7223c75dbb14f38558e62487409175 not found: ID does not exist" Jul 28 11:22:08.598645 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.598624 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:22:08.614898 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.614877 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-0a6eb5-5b8c4cf578-dbd5s"] Jul 28 11:22:08.638008 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.637985 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:22:08.648751 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.648717 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0a6eb5-67f6dd964d-89sb8"] Jul 28 11:22:08.667635 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.667610 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:22:08.679093 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:08.679070 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0a6eb5-c5c9f5784-6qb67"] Jul 28 11:22:09.374180 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:09.374149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:09.388565 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:09.388537 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b1345d-e571-4310-aba4-a8bb70a8ab33" path="/var/lib/kubelet/pods/69b1345d-e571-4310-aba4-a8bb70a8ab33/volumes" Jul 28 11:22:09.388936 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:09.388922 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06587de-95e0-4668-923d-ec6141f9e91e" path="/var/lib/kubelet/pods/d06587de-95e0-4668-923d-ec6141f9e91e/volumes" Jul 28 11:22:09.389262 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:09.389249 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea75d814-724b-4d65-9adb-ce718849eda7" path="/var/lib/kubelet/pods/ea75d814-724b-4d65-9adb-ce718849eda7/volumes" Jul 28 11:22:09.849299 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:09.849268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:18.835197 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:18.835162 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:20.322611 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:20.322581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:42.758119 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.758080 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:22:42.758579 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.758354 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" podUID="8d4025dc-a75b-4a8b-af45-bc374d294e37" containerName="maas-api" containerID="cri-o://d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370" gracePeriod=30 Jul 28 11:22:42.799268 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.799239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:42.821340 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.821315 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:22:42.821561 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.821536 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" podUID="8ded5521-3f39-4c42-ba24-2482687236f6" containerName="payload-processing" containerID="cri-o://df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0" gracePeriod=30 Jul 28 11:22:42.826085 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.826058 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:22:42.826303 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.826281 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" podUID="5775b75d-2572-4ae4-9aab-f7b015a348b0" containerName="payload-pre-processing" containerID="cri-o://6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6" gracePeriod=30 Jul 28 11:22:42.895086 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:42.895056 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:43.005011 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.004986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:43.091805 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.091780 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:22:43.124754 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.124705 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:22:43.135446 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.135418 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:43.142510 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.142491 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:22:43.220341 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220311 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") pod \"8d4025dc-a75b-4a8b-af45-bc374d294e37\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " Jul 28 11:22:43.220482 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86r5s\" (UniqueName: \"kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s\") pod \"5775b75d-2572-4ae4-9aab-f7b015a348b0\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " Jul 28 11:22:43.220482 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220441 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q7pc\" (UniqueName: \"kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc\") pod \"8d4025dc-a75b-4a8b-af45-bc374d294e37\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " Jul 28 11:22:43.220482 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220466 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume\") pod \"5775b75d-2572-4ae4-9aab-f7b015a348b0\" (UID: \"5775b75d-2572-4ae4-9aab-f7b015a348b0\") " Jul 28 11:22:43.220647 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220497 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2ktl\" (UniqueName: \"kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl\") pod \"8ded5521-3f39-4c42-ba24-2482687236f6\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " Jul 28 11:22:43.220647 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220588 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume\") pod \"8ded5521-3f39-4c42-ba24-2482687236f6\" (UID: \"8ded5521-3f39-4c42-ba24-2482687236f6\") " Jul 28 11:22:43.220647 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220626 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca\") pod \"8d4025dc-a75b-4a8b-af45-bc374d294e37\" (UID: \"8d4025dc-a75b-4a8b-af45-bc374d294e37\") " Jul 28 11:22:43.221031 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.220987 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5775b75d-2572-4ae4-9aab-f7b015a348b0" (UID: "5775b75d-2572-4ae4-9aab-f7b015a348b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:43.221254 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.221226 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d4025dc-a75b-4a8b-af45-bc374d294e37" (UID: "8d4025dc-a75b-4a8b-af45-bc374d294e37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:43.221360 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.221030 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ded5521-3f39-4c42-ba24-2482687236f6" (UID: "8ded5521-3f39-4c42-ba24-2482687236f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:22:43.222647 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.222629 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d4025dc-a75b-4a8b-af45-bc374d294e37" (UID: "8d4025dc-a75b-4a8b-af45-bc374d294e37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:22:43.222727 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.222655 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc" (OuterVolumeSpecName: "kube-api-access-4q7pc") pod "8d4025dc-a75b-4a8b-af45-bc374d294e37" (UID: "8d4025dc-a75b-4a8b-af45-bc374d294e37"). InnerVolumeSpecName "kube-api-access-4q7pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:43.223206 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.223186 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s" (OuterVolumeSpecName: "kube-api-access-86r5s") pod "5775b75d-2572-4ae4-9aab-f7b015a348b0" (UID: "5775b75d-2572-4ae4-9aab-f7b015a348b0"). InnerVolumeSpecName "kube-api-access-86r5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:43.223264 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.223206 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl" (OuterVolumeSpecName: "kube-api-access-q2ktl") pod "8ded5521-3f39-4c42-ba24-2482687236f6" (UID: "8ded5521-3f39-4c42-ba24-2482687236f6"). InnerVolumeSpecName "kube-api-access-q2ktl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322187 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86r5s\" (UniqueName: \"kubernetes.io/projected/5775b75d-2572-4ae4-9aab-f7b015a348b0-kube-api-access-86r5s\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322208 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q7pc\" (UniqueName: \"kubernetes.io/projected/8d4025dc-a75b-4a8b-af45-bc374d294e37-kube-api-access-4q7pc\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322219 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5775b75d-2572-4ae4-9aab-f7b015a348b0-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322230 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ktl\" (UniqueName: \"kubernetes.io/projected/8ded5521-3f39-4c42-ba24-2482687236f6-kube-api-access-q2ktl\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322238 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ded5521-3f39-4c42-ba24-2482687236f6-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322247 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4025dc-a75b-4a8b-af45-bc374d294e37-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.322486 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.322256 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4025dc-a75b-4a8b-af45-bc374d294e37-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:22:43.613243 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.613142 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:22:43.754618 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.754585 2567 generic.go:356] "Generic (PLEG): container finished" podID="5775b75d-2572-4ae4-9aab-f7b015a348b0" containerID="6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6" exitCode=0 Jul 28 11:22:43.754821 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.754669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" event={"ID":"5775b75d-2572-4ae4-9aab-f7b015a348b0","Type":"ContainerDied","Data":"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6"} Jul 28 11:22:43.754821 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.754685 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" Jul 28 11:22:43.754821 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.754723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m" event={"ID":"5775b75d-2572-4ae4-9aab-f7b015a348b0","Type":"ContainerDied","Data":"ba6e99959717520ef901b7f89c69d246a3c9b3fa777c931bed4e0f6114196d54"} Jul 28 11:22:43.754821 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.754770 2567 scope.go:117] "RemoveContainer" containerID="6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6" Jul 28 11:22:43.756178 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.756153 2567 generic.go:356] "Generic (PLEG): container finished" podID="8ded5521-3f39-4c42-ba24-2482687236f6" containerID="df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0" exitCode=0 Jul 28 11:22:43.756310 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.756215 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" Jul 28 11:22:43.756520 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.756243 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" event={"ID":"8ded5521-3f39-4c42-ba24-2482687236f6","Type":"ContainerDied","Data":"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0"} Jul 28 11:22:43.756595 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.756553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2" event={"ID":"8ded5521-3f39-4c42-ba24-2482687236f6","Type":"ContainerDied","Data":"83a4607680aac3fa2d23fcbc19a10dcb36356119034710575772de0ab1399038"} Jul 28 11:22:43.757926 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.757897 2567 generic.go:356] "Generic (PLEG): container finished" podID="8d4025dc-a75b-4a8b-af45-bc374d294e37" containerID="d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370" exitCode=0 Jul 28 11:22:43.758042 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.757950 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" Jul 28 11:22:43.758042 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.757949 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" event={"ID":"8d4025dc-a75b-4a8b-af45-bc374d294e37","Type":"ContainerDied","Data":"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370"} Jul 28 11:22:43.758159 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.758058 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p" event={"ID":"8d4025dc-a75b-4a8b-af45-bc374d294e37","Type":"ContainerDied","Data":"b6826f09bbbb705922b3510d053efccbb1819331ace78445745977b68c07f364"} Jul 28 11:22:43.765402 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.765219 2567 scope.go:117] "RemoveContainer" containerID="6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6" Jul 28 11:22:43.765522 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:43.765503 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6\": container with ID starting with 6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6 not found: ID does not exist" containerID="6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6" Jul 28 11:22:43.765593 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.765529 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6"} err="failed to get container status \"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6\": rpc error: code = NotFound desc = could not find container \"6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6\": container with ID starting with 6c30f29896017009d6a4ec86655fef1b32ee25eae86688bbf18c5d1107811bc6 not found: ID does not exist" Jul 28 11:22:43.765593 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.765545 2567 scope.go:117] "RemoveContainer" containerID="df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0" Jul 28 11:22:43.776607 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.776534 2567 scope.go:117] "RemoveContainer" containerID="df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0" Jul 28 11:22:43.776970 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:43.776947 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0\": container with ID starting with df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0 not found: ID does not exist" containerID="df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0" Jul 28 11:22:43.777058 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.776978 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0"} err="failed to get container status \"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0\": rpc error: code = NotFound desc = could not find container \"df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0\": container with ID starting with df7e5d1028613a19c8005be97fca77909d3772808bffde1078e2e259410a5bd0 not found: ID does not exist" Jul 28 11:22:43.777058 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.776999 2567 scope.go:117] "RemoveContainer" containerID="d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370" Jul 28 11:22:43.779795 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.779776 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:22:43.784718 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.784693 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1c9694-79dccb5876-cqp7m"] Jul 28 11:22:43.785526 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.785505 2567 scope.go:117] "RemoveContainer" containerID="d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370" Jul 28 11:22:43.785800 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:22:43.785780 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370\": container with ID starting with d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370 not found: ID does not exist" containerID="d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370" Jul 28 11:22:43.785900 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.785805 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370"} err="failed to get container status \"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370\": rpc error: code = NotFound desc = could not find container \"d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370\": container with ID starting with d790fde66dd1fd8f547f30b01a97fb8c27b2c94e91b4bb844041c79c99417370 not found: ID does not exist" Jul 28 11:22:43.797125 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.797106 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:22:43.804962 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.804937 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1c9694-7c947b764d-28g2p"] Jul 28 11:22:43.817073 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.817054 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:22:43.821775 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:43.821747 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c9694-776dc85f4-ctpm2"] Jul 28 11:22:45.387360 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:45.387328 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5775b75d-2572-4ae4-9aab-f7b015a348b0" path="/var/lib/kubelet/pods/5775b75d-2572-4ae4-9aab-f7b015a348b0/volumes" Jul 28 11:22:45.387761 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:45.387696 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4025dc-a75b-4a8b-af45-bc374d294e37" path="/var/lib/kubelet/pods/8d4025dc-a75b-4a8b-af45-bc374d294e37/volumes" Jul 28 11:22:45.388024 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:45.388010 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ded5521-3f39-4c42-ba24-2482687236f6" path="/var/lib/kubelet/pods/8ded5521-3f39-4c42-ba24-2482687236f6/volumes" Jul 28 11:22:48.643716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:22:48.643679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:24.797388 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:24.797358 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:26.425981 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:26.425933 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:32.779008 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:32.778976 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:32.853599 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:32.853563 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:33.132434 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:33.132349 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:33.439011 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:33.438922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:45.251557 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.251514 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:23:45.252026 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.251793 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" podUID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" containerName="maas-api" containerID="cri-o://64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd" gracePeriod=30 Jul 28 11:23:45.290958 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.290935 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:45.331086 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.331056 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:23:45.331310 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.331273 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" podUID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" containerName="payload-pre-processing" containerID="cri-o://7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941" gracePeriod=30 Jul 28 11:23:45.390681 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.390643 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:45.494272 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.493778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:45.584526 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.584504 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:23:45.619616 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.619589 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:23:45.692472 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692447 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4t6r\" (UniqueName: \"kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r\") pod \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " Jul 28 11:23:45.692599 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692481 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca\") pod \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " Jul 28 11:23:45.692599 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692511 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume\") pod \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " Jul 28 11:23:45.692599 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692562 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz5bk\" (UniqueName: \"kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk\") pod \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\" (UID: \"ab3ab7f4-e571-48fd-bc34-acd625e4910c\") " Jul 28 11:23:45.692724 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692599 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") pod \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\" (UID: \"9fd95959-4371-4e9e-9aaf-a675d2a341f8\") " Jul 28 11:23:45.692949 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692920 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fd95959-4371-4e9e-9aaf-a675d2a341f8" (UID: "9fd95959-4371-4e9e-9aaf-a675d2a341f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:23:45.693022 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.692930 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab3ab7f4-e571-48fd-bc34-acd625e4910c" (UID: "ab3ab7f4-e571-48fd-bc34-acd625e4910c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:23:45.694930 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.694899 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fd95959-4371-4e9e-9aaf-a675d2a341f8" (UID: "9fd95959-4371-4e9e-9aaf-a675d2a341f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:23:45.695031 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.694983 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk" (OuterVolumeSpecName: "kube-api-access-rz5bk") pod "ab3ab7f4-e571-48fd-bc34-acd625e4910c" (UID: "ab3ab7f4-e571-48fd-bc34-acd625e4910c"). InnerVolumeSpecName "kube-api-access-rz5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:23:45.695031 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.695017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r" (OuterVolumeSpecName: "kube-api-access-t4t6r") pod "9fd95959-4371-4e9e-9aaf-a675d2a341f8" (UID: "9fd95959-4371-4e9e-9aaf-a675d2a341f8"). InnerVolumeSpecName "kube-api-access-t4t6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:23:45.793825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.793760 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4t6r\" (UniqueName: \"kubernetes.io/projected/9fd95959-4371-4e9e-9aaf-a675d2a341f8-kube-api-access-t4t6r\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:23:45.793825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.793782 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd95959-4371-4e9e-9aaf-a675d2a341f8-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:23:45.793825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.793791 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3ab7f4-e571-48fd-bc34-acd625e4910c-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:23:45.793825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.793800 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5bk\" (UniqueName: \"kubernetes.io/projected/ab3ab7f4-e571-48fd-bc34-acd625e4910c-kube-api-access-rz5bk\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:23:45.793825 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:45.793808 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd95959-4371-4e9e-9aaf-a675d2a341f8-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:23:46.103443 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.103364 2567 generic.go:356] "Generic (PLEG): container finished" podID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" containerID="7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941" exitCode=0 Jul 28 11:23:46.103443 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.103406 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" Jul 28 11:23:46.103656 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.103448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" event={"ID":"ab3ab7f4-e571-48fd-bc34-acd625e4910c","Type":"ContainerDied","Data":"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941"} Jul 28 11:23:46.103656 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.103497 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688" event={"ID":"ab3ab7f4-e571-48fd-bc34-acd625e4910c","Type":"ContainerDied","Data":"1992ca91979203a9f8197bd4e85c93af89b02e1b9a831159c1b8df7753d9cb97"} Jul 28 11:23:46.103656 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.103516 2567 scope.go:117] "RemoveContainer" containerID="7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941" Jul 28 11:23:46.104850 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.104826 2567 generic.go:356] "Generic (PLEG): container finished" podID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" containerID="64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd" exitCode=0 Jul 28 11:23:46.104938 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.104882 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" Jul 28 11:23:46.104938 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.104903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" event={"ID":"9fd95959-4371-4e9e-9aaf-a675d2a341f8","Type":"ContainerDied","Data":"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd"} Jul 28 11:23:46.105058 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.104954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk" event={"ID":"9fd95959-4371-4e9e-9aaf-a675d2a341f8","Type":"ContainerDied","Data":"18c8e0706802e4d990d6a88097335c50c93639d2c7ec2599f45d64e42b79a6ea"} Jul 28 11:23:46.113377 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.113357 2567 scope.go:117] "RemoveContainer" containerID="7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941" Jul 28 11:23:46.113675 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:23:46.113656 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941\": container with ID starting with 7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941 not found: ID does not exist" containerID="7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941" Jul 28 11:23:46.113797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.113686 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941"} err="failed to get container status \"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941\": rpc error: code = NotFound desc = could not find container \"7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941\": container with ID starting with 7b3abc8a58624bff45d6db9ad9c5d4d19fe0ee8cb19fb3656f2358d64669a941 not found: ID does not exist" Jul 28 11:23:46.113797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.113708 2567 scope.go:117] "RemoveContainer" containerID="64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd" Jul 28 11:23:46.122266 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.122251 2567 scope.go:117] "RemoveContainer" containerID="64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd" Jul 28 11:23:46.122524 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:23:46.122508 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd\": container with ID starting with 64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd not found: ID does not exist" containerID="64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd" Jul 28 11:23:46.122576 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.122531 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd"} err="failed to get container status \"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd\": rpc error: code = NotFound desc = could not find container \"64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd\": container with ID starting with 64b703182bcfec22b76298e035c56bf142750ba842932b6e40d485f4f269bcbd not found: ID does not exist" Jul 28 11:23:46.134993 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.134973 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:23:46.137371 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.137341 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a93417-6bc8dc4548-lc688"] Jul 28 11:23:46.155639 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.155617 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:23:46.159703 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:46.159681 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a93417-6dc6df68b8-xwpbk"] Jul 28 11:23:47.389598 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:47.389553 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" path="/var/lib/kubelet/pods/9fd95959-4371-4e9e-9aaf-a675d2a341f8/volumes" Jul 28 11:23:47.390168 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:47.390144 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" path="/var/lib/kubelet/pods/ab3ab7f4-e571-48fd-bc34-acd625e4910c/volumes" Jul 28 11:23:48.669956 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:48.669921 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:49.071433 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:49.071399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:55.793211 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:55.793176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:55.902121 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:55.902090 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:56.072722 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:56.072635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:23:56.344827 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:23:56.344722 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:02.597525 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:02.597493 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:02.698017 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:02.697986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:07.780633 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:07.780597 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:24:07.781141 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:07.780821 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" podUID="d5e8a39f-2b4a-4017-b6fb-b83061762610" containerName="maas-api" containerID="cri-o://35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751" gracePeriod=30 Jul 28 11:24:07.812278 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:07.812239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:07.870821 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:07.870793 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:24:07.960899 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:07.960828 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:08.049268 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.049244 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:24:08.060496 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.060475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:08.088042 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.088011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls\") pod \"d5e8a39f-2b4a-4017-b6fb-b83061762610\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " Jul 28 11:24:08.088195 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.088084 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca\") pod \"d5e8a39f-2b4a-4017-b6fb-b83061762610\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " Jul 28 11:24:08.088195 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.088182 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrmgr\" (UniqueName: \"kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr\") pod \"d5e8a39f-2b4a-4017-b6fb-b83061762610\" (UID: \"d5e8a39f-2b4a-4017-b6fb-b83061762610\") " Jul 28 11:24:08.088432 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.088409 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5e8a39f-2b4a-4017-b6fb-b83061762610" (UID: "d5e8a39f-2b4a-4017-b6fb-b83061762610"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:24:08.088599 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.088582 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e8a39f-2b4a-4017-b6fb-b83061762610-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:08.090407 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.090382 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5e8a39f-2b4a-4017-b6fb-b83061762610" (UID: "d5e8a39f-2b4a-4017-b6fb-b83061762610"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:24:08.090504 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.090434 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr" (OuterVolumeSpecName: "kube-api-access-hrmgr") pod "d5e8a39f-2b4a-4017-b6fb-b83061762610" (UID: "d5e8a39f-2b4a-4017-b6fb-b83061762610"). InnerVolumeSpecName "kube-api-access-hrmgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:24:08.189454 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.189425 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e8a39f-2b4a-4017-b6fb-b83061762610-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:08.189454 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.189451 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrmgr\" (UniqueName: \"kubernetes.io/projected/d5e8a39f-2b4a-4017-b6fb-b83061762610-kube-api-access-hrmgr\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:08.248096 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248060 2567 generic.go:356] "Generic (PLEG): container finished" podID="d5e8a39f-2b4a-4017-b6fb-b83061762610" containerID="35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751" exitCode=0 Jul 28 11:24:08.248246 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248133 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" Jul 28 11:24:08.248246 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" event={"ID":"d5e8a39f-2b4a-4017-b6fb-b83061762610","Type":"ContainerDied","Data":"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751"} Jul 28 11:24:08.248246 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq" event={"ID":"d5e8a39f-2b4a-4017-b6fb-b83061762610","Type":"ContainerDied","Data":"7a73031392a803ae67897f177699c466f8b7f0eb47cf23b5358763a83889ff92"} Jul 28 11:24:08.248407 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248270 2567 scope.go:117] "RemoveContainer" containerID="35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751" Jul 28 11:24:08.248531 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.248500 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" podUID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" containerName="payload-pre-processing" containerID="cri-o://1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5" gracePeriod=30 Jul 28 11:24:08.259182 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.259164 2567 scope.go:117] "RemoveContainer" containerID="35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751" Jul 28 11:24:08.259441 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:24:08.259422 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751\": container with ID starting with 35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751 not found: ID does not exist" containerID="35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751" Jul 28 11:24:08.259514 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.259452 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751"} err="failed to get container status \"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751\": rpc error: code = NotFound desc = could not find container \"35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751\": container with ID starting with 35209731e5287335dcd201e853557b2ddb007e2a90aef526ebefedc38118e751 not found: ID does not exist" Jul 28 11:24:08.296158 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.296110 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:24:08.298285 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.298267 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4cda06-75544ddfc6-zztvq"] Jul 28 11:24:08.492619 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.492595 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:24:08.593823 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.593730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbtgc\" (UniqueName: \"kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc\") pod \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " Jul 28 11:24:08.593823 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.593791 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume\") pod \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\" (UID: \"a55a1e8c-bce5-4462-a3ac-f042e65ae232\") " Jul 28 11:24:08.594125 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.594104 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a55a1e8c-bce5-4462-a3ac-f042e65ae232" (UID: "a55a1e8c-bce5-4462-a3ac-f042e65ae232"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:24:08.596004 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.595981 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc" (OuterVolumeSpecName: "kube-api-access-fbtgc") pod "a55a1e8c-bce5-4462-a3ac-f042e65ae232" (UID: "a55a1e8c-bce5-4462-a3ac-f042e65ae232"). InnerVolumeSpecName "kube-api-access-fbtgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:24:08.695110 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.695083 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbtgc\" (UniqueName: \"kubernetes.io/projected/a55a1e8c-bce5-4462-a3ac-f042e65ae232-kube-api-access-fbtgc\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:08.695110 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:08.695106 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55a1e8c-bce5-4462-a3ac-f042e65ae232-plugins-config-volume\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:09.255509 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.255466 2567 generic.go:356] "Generic (PLEG): container finished" podID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" containerID="1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5" exitCode=0 Jul 28 11:24:09.256034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.255514 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" Jul 28 11:24:09.256034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.255553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" event={"ID":"a55a1e8c-bce5-4462-a3ac-f042e65ae232","Type":"ContainerDied","Data":"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5"} Jul 28 11:24:09.256034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.255608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w" event={"ID":"a55a1e8c-bce5-4462-a3ac-f042e65ae232","Type":"ContainerDied","Data":"a4b6d416046e2158eabe242aef26bf470f27ff0ae35a1d0d2eeec94c9eda0dda"} Jul 28 11:24:09.256034 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.255632 2567 scope.go:117] "RemoveContainer" containerID="1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5" Jul 28 11:24:09.265059 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.265044 2567 scope.go:117] "RemoveContainer" containerID="1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5" Jul 28 11:24:09.265325 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:24:09.265298 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5\": container with ID starting with 1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5 not found: ID does not exist" containerID="1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5" Jul 28 11:24:09.265398 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.265324 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5"} err="failed to get container status \"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5\": rpc error: code = NotFound desc = could not find container \"1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5\": container with ID starting with 1506006d945db40993f4dc5b3c4c4d5ac87545f3e0ca06ec9853eac918b224f5 not found: ID does not exist" Jul 28 11:24:09.278505 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.278483 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:24:09.282297 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.282276 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cda06-697fb69b-sjg4w"] Jul 28 11:24:09.388286 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.388261 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" path="/var/lib/kubelet/pods/a55a1e8c-bce5-4462-a3ac-f042e65ae232/volumes" Jul 28 11:24:09.388703 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:09.388688 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e8a39f-2b4a-4017-b6fb-b83061762610" path="/var/lib/kubelet/pods/d5e8a39f-2b4a-4017-b6fb-b83061762610/volumes" Jul 28 11:24:11.584434 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:11.584384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:12.027441 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:12.027411 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:18.492123 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:18.492085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:18.592893 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:18.592847 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:18.811330 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:18.811241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:19.221000 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:19.220952 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:30.915576 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:30.915545 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:24:30.916082 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:30.915769 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" podUID="380b5377-52fe-4f19-9266-aad48685dce0" containerName="maas-api" containerID="cri-o://9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5" gracePeriod=30 Jul 28 11:24:30.949995 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:30.949972 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:31.067290 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.067258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:31.137019 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.136980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:31.190699 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.190676 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:24:31.300554 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.300524 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thzll\" (UniqueName: \"kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll\") pod \"380b5377-52fe-4f19-9266-aad48685dce0\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " Jul 28 11:24:31.300716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.300598 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca\") pod \"380b5377-52fe-4f19-9266-aad48685dce0\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " Jul 28 11:24:31.300716 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.300619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") pod \"380b5377-52fe-4f19-9266-aad48685dce0\" (UID: \"380b5377-52fe-4f19-9266-aad48685dce0\") " Jul 28 11:24:31.301043 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.301017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "380b5377-52fe-4f19-9266-aad48685dce0" (UID: "380b5377-52fe-4f19-9266-aad48685dce0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 11:24:31.302827 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.302793 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll" (OuterVolumeSpecName: "kube-api-access-thzll") pod "380b5377-52fe-4f19-9266-aad48685dce0" (UID: "380b5377-52fe-4f19-9266-aad48685dce0"). InnerVolumeSpecName "kube-api-access-thzll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 11:24:31.302951 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.302877 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "380b5377-52fe-4f19-9266-aad48685dce0" (UID: "380b5377-52fe-4f19-9266-aad48685dce0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 11:24:31.380528 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.380493 2567 generic.go:356] "Generic (PLEG): container finished" podID="380b5377-52fe-4f19-9266-aad48685dce0" containerID="9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5" exitCode=0 Jul 28 11:24:31.380657 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.380611 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" Jul 28 11:24:31.387704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.387676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" event={"ID":"380b5377-52fe-4f19-9266-aad48685dce0","Type":"ContainerDied","Data":"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5"} Jul 28 11:24:31.387822 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.387728 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg" event={"ID":"380b5377-52fe-4f19-9266-aad48685dce0","Type":"ContainerDied","Data":"ab1fc2701d025f5eb93e09cd779f4c8be5e040aeb36b1fed9d20d0cede800c21"} Jul 28 11:24:31.387822 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.387796 2567 scope.go:117] "RemoveContainer" containerID="9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5" Jul 28 11:24:31.396704 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.396685 2567 scope.go:117] "RemoveContainer" containerID="9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5" Jul 28 11:24:31.397035 ip-10-0-130-55 kubenswrapper[2567]: E0728 11:24:31.397014 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5\": container with ID starting with 9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5 not found: ID does not exist" containerID="9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5" Jul 28 11:24:31.397091 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.397046 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5"} err="failed to get container status \"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5\": rpc error: code = NotFound desc = could not find container \"9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5\": container with ID starting with 9cde747210986fdf5f632004bffe87c5070c0acc83b9183c03c04cbdbafa1af5 not found: ID does not exist" Jul 28 11:24:31.401482 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.401465 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b5377-52fe-4f19-9266-aad48685dce0-openshift-service-ca\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:31.401549 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.401485 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b5377-52fe-4f19-9266-aad48685dce0-maas-api-tls\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:31.401549 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.401494 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzll\" (UniqueName: \"kubernetes.io/projected/380b5377-52fe-4f19-9266-aad48685dce0-kube-api-access-thzll\") on node \"ip-10-0-130-55.ec2.internal\" DevicePath \"\"" Jul 28 11:24:31.412305 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.412275 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:24:31.420668 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:31.420611 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-683bdc-6d4cd656bf-4jlxg"] Jul 28 11:24:33.386706 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:33.386676 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380b5377-52fe-4f19-9266-aad48685dce0" path="/var/lib/kubelet/pods/380b5377-52fe-4f19-9266-aad48685dce0/volumes" Jul 28 11:24:34.350652 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:34.350615 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:34.876258 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:34.876222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:40.885697 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:40.885669 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:41.709132 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:41.709101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:41.840556 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:41.840519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:42.052648 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:42.052576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:42.276597 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:42.276562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:24:49.252269 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:49.252168 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d7d4f6df-67vlx_9f933be8-6456-4d3f-928c-af17892ee591/manager/0.log" Jul 28 11:24:49.812880 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:49.812847 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-v2b8k_47faa125-f9cf-4372-8e33-e2b5d6fc2b39/copy-manifests/0.log" Jul 28 11:24:49.828089 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:49.828059 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-v2b8k_47faa125-f9cf-4372-8e33-e2b5d6fc2b39/manager/0.log" Jul 28 11:24:50.365302 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:50.365275 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77c87dc85b-7zvrf_2a8e01ef-9e9c-45da-a673-acbeeaa7b2c9/maas-api/0.log" Jul 28 11:24:50.877915 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:50.877890 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lbbqs_9a2ce82a-d33d-4a36-8810-3b4e56e56b87/postgres/0.log" Jul 28 11:24:51.671432 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.671404 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/pull/0.log" Jul 28 11:24:51.678651 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.678625 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/extract/0.log" Jul 28 11:24:51.685052 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.685030 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/util/0.log" Jul 28 11:24:51.832933 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.832912 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/util/0.log" Jul 28 11:24:51.839765 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.839727 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/pull/0.log" Jul 28 11:24:51.846655 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.846638 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/extract/0.log" Jul 28 11:24:51.963122 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.963100 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/pull/0.log" Jul 28 11:24:51.969836 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.969820 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/extract/0.log" Jul 28 11:24:51.976429 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:51.976412 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/util/0.log" Jul 28 11:24:52.093923 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:52.093903 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/util/0.log" Jul 28 11:24:52.103106 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:52.103091 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/pull/0.log" Jul 28 11:24:52.115684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:52.115666 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/extract/0.log" Jul 28 11:24:53.645847 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:53.645815 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w2lzb_11e63ffc-3903-4e58-ba63-0ceac4f58f4c/discovery/0.log" Jul 28 11:24:54.168268 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:54.168237 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v57vq_2171fb23-48ec-41f9-a64d-a56007c59f83/payload-pre-processing/0.log" Jul 28 11:24:54.293512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:54.293474 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log" Jul 28 11:24:54.793287 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:54.793263 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9mqsx_8dae8d03-a402-49d1-b9d9-813a8e49420f/main/0.log" Jul 28 11:24:55.292244 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:24:55.292212 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bpm9f_13a14c06-f593-49f5-99b4-7357bdd738c6/main/0.log" Jul 28 11:25:01.295271 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:01.295236 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:25:01.461553 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:01.461515 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:25:04.420679 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:04.420647 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-swstv_e67fa4a7-0013-40fb-b97b-9ab58fa62120/global-pull-secret-syncer/0.log" Jul 28 11:25:04.492961 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:04.492936 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" Jul 28 11:25:04.640980 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:04.640951 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-55.ec2.internal_ce85c6809f8a3cdc882d4b65d5c5ac9d/haproxy/0.log" Jul 28 11:25:09.055218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.055186 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/extract/0.log" Jul 28 11:25:09.086178 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.086148 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/util/0.log" Jul 28 11:25:09.119771 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.119726 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759f6snr_40f011ac-ad41-4f85-ab22-b760f188eb9b/pull/0.log" Jul 28 11:25:09.172360 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.172336 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/extract/0.log" Jul 28 11:25:09.200715 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.200697 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/util/0.log" Jul 28 11:25:09.225962 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.225939 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d6nwr_d3ed5711-83b2-4d9c-96e9-f637d213f034/pull/0.log" Jul 28 11:25:09.254797 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.254776 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/extract/0.log" Jul 28 11:25:09.284483 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.284462 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/util/0.log" Jul 28 11:25:09.311991 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.311939 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l4vdk_b83ae250-366c-4b93-ba4d-843c38ac2eb7/pull/0.log" Jul 28 11:25:09.345477 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.345453 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/extract/0.log" Jul 28 11:25:09.372951 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.372929 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/util/0.log" Jul 28 11:25:09.401371 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:09.401354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16jmsx_936f30cc-3c31-4250-9f79-3996a708ed83/pull/0.log" Jul 28 11:25:12.160807 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.160709 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/alertmanager/0.log" Jul 28 11:25:12.188314 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.188290 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/config-reloader/0.log" Jul 28 11:25:12.224783 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.224761 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/kube-rbac-proxy-web/0.log" Jul 28 11:25:12.249512 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.249489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/kube-rbac-proxy/0.log" Jul 28 11:25:12.274503 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.274476 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/kube-rbac-proxy-metric/0.log" Jul 28 11:25:12.299045 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.299025 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/prom-label-proxy/0.log" Jul 28 11:25:12.326079 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.326061 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f8cdfd6-c6c2-4772-949b-0dcfc785cc2f/init-config-reloader/0.log" Jul 28 11:25:12.417424 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.417373 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h82g2_a87b2508-ae9d-47aa-a853-80eab2d5379e/kube-state-metrics/0.log" Jul 28 11:25:12.437291 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.437271 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h82g2_a87b2508-ae9d-47aa-a853-80eab2d5379e/kube-rbac-proxy-main/0.log" Jul 28 11:25:12.466805 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.466784 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h82g2_a87b2508-ae9d-47aa-a853-80eab2d5379e/kube-rbac-proxy-self/0.log" Jul 28 11:25:12.492906 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.492885 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77797f6788-v5lvd_d30e430f-8bdf-4a42-94f8-915ea38d72c0/metrics-server/0.log" Jul 28 11:25:12.637885 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.637863 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cntf4_76063a07-970e-401b-ac66-c37b3c1d206b/node-exporter/0.log" Jul 28 11:25:12.659949 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.659929 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cntf4_76063a07-970e-401b-ac66-c37b3c1d206b/kube-rbac-proxy/0.log" Jul 28 11:25:12.680720 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:12.680670 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cntf4_76063a07-970e-401b-ac66-c37b3c1d206b/init-textfile/0.log" Jul 28 11:25:13.199592 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.199561 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b95489946-nl589_fc0c17b7-a505-4bf1-811a-bf63d9f9436f/telemeter-client/0.log" Jul 28 11:25:13.221941 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.221918 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b95489946-nl589_fc0c17b7-a505-4bf1-811a-bf63d9f9436f/reload/0.log" Jul 28 11:25:13.243184 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.243122 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b95489946-nl589_fc0c17b7-a505-4bf1-811a-bf63d9f9436f/kube-rbac-proxy/0.log" Jul 28 11:25:13.270251 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.270228 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/thanos-query/0.log" Jul 28 11:25:13.303045 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.303024 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/kube-rbac-proxy-web/0.log" Jul 28 11:25:13.323671 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.323651 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/kube-rbac-proxy/0.log" Jul 28 11:25:13.347525 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.347509 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/prom-label-proxy/0.log" Jul 28 11:25:13.370521 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.370499 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/kube-rbac-proxy-rules/0.log" Jul 28 11:25:13.393174 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:13.393153 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64646cb4fb-fntvg_9d4752ab-620c-4749-b5ea-bcc867ff6e8c/kube-rbac-proxy-metrics/0.log" Jul 28 11:25:14.385450 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.385420 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm"] Jul 28 11:25:14.386012 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.385988 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e8a39f-2b4a-4017-b6fb-b83061762610" containerName="maas-api" Jul 28 11:25:14.386145 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386130 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e8a39f-2b4a-4017-b6fb-b83061762610" containerName="maas-api" Jul 28 11:25:14.386218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386175 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4025dc-a75b-4a8b-af45-bc374d294e37" containerName="maas-api" Jul 28 11:25:14.386218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386185 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4025dc-a75b-4a8b-af45-bc374d294e37" containerName="maas-api" Jul 28 11:25:14.386218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" containerName="maas-api" Jul 28 11:25:14.386218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386203 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" containerName="maas-api" Jul 28 11:25:14.386218 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386217 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386226 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386235 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ded5521-3f39-4c42-ba24-2482687236f6" containerName="payload-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386244 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ded5521-3f39-4c42-ba24-2482687236f6" containerName="payload-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386258 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380b5377-52fe-4f19-9266-aad48685dce0" containerName="maas-api" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386266 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="380b5377-52fe-4f19-9266-aad48685dce0" containerName="maas-api" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386280 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea75d814-724b-4d65-9adb-ce718849eda7" containerName="maas-api" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386288 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea75d814-724b-4d65-9adb-ce718849eda7" containerName="maas-api" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386302 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5775b75d-2572-4ae4-9aab-f7b015a348b0" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386311 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5775b75d-2572-4ae4-9aab-f7b015a348b0" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386322 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b1345d-e571-4310-aba4-a8bb70a8ab33" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386330 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b1345d-e571-4310-aba4-a8bb70a8ab33" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386343 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386351 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" containerName="payload-pre-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386362 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06587de-95e0-4668-923d-ec6141f9e91e" containerName="payload-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386370 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06587de-95e0-4668-923d-ec6141f9e91e" containerName="payload-processing" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386465 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea75d814-724b-4d65-9adb-ce718849eda7" containerName="maas-api" Jul 28 11:25:14.386473 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386478 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55a1e8c-bce5-4462-a3ac-f042e65ae232" containerName="payload-pre-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386488 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ded5521-3f39-4c42-ba24-2482687236f6" containerName="payload-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386498 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b1345d-e571-4310-aba4-a8bb70a8ab33" containerName="payload-pre-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386507 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5775b75d-2572-4ae4-9aab-f7b015a348b0" containerName="payload-pre-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386514 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="380b5377-52fe-4f19-9266-aad48685dce0" containerName="maas-api" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386524 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06587de-95e0-4668-923d-ec6141f9e91e" containerName="payload-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386534 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e8a39f-2b4a-4017-b6fb-b83061762610" containerName="maas-api" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386543 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4025dc-a75b-4a8b-af45-bc374d294e37" containerName="maas-api" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386551 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3ab7f4-e571-48fd-bc34-acd625e4910c" containerName="payload-pre-processing" Jul 28 11:25:14.387351 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.386560 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd95959-4371-4e9e-9aaf-a675d2a341f8" containerName="maas-api" Jul 28 11:25:14.390088 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.390068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.393116 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.393097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgn8k\"/\"openshift-service-ca.crt\"" Jul 28 11:25:14.393220 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.393101 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zgn8k\"/\"default-dockercfg-j5kdx\"" Jul 28 11:25:14.394146 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.394110 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgn8k\"/\"kube-root-ca.crt\"" Jul 28 11:25:14.399374 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.399351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm"] Jul 28 11:25:14.442179 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.442153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-podres\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.442275 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.442191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jdg\" (UniqueName: \"kubernetes.io/projected/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-kube-api-access-k4jdg\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.442275 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.442212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-proc\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.442445 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.442419 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-sys\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.442615 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.442592 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-lib-modules\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543477 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-lib-modules\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-podres\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jdg\" (UniqueName: \"kubernetes.io/projected/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-kube-api-access-k4jdg\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543572 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-proc\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543608 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-podres\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-lib-modules\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-sys\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543684 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-sys\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.543884 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.543695 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-proc\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.553053 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.553033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jdg\" (UniqueName: \"kubernetes.io/projected/07d96ff7-f6f9-4205-a00e-40d184ecbeb1-kube-api-access-k4jdg\") pod \"perf-node-gather-daemonset-74cfm\" (UID: \"07d96ff7-f6f9-4205-a00e-40d184ecbeb1\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.700667 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.700613 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:14.835856 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:14.834434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm"] Jul 28 11:25:15.611408 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:15.611370 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" event={"ID":"07d96ff7-f6f9-4205-a00e-40d184ecbeb1","Type":"ContainerStarted","Data":"90d91b2386647943e9b0cffddd077c90e5642a992801f61afec5d9fd51da69bd"} Jul 28 11:25:15.611816 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:15.611427 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" event={"ID":"07d96ff7-f6f9-4205-a00e-40d184ecbeb1","Type":"ContainerStarted","Data":"25a7cc24ed49d23ea7c6a2acecdbb2c28c82c0f5dd66f5d96e2de472ef496657"} Jul 28 11:25:15.611816 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:15.611449 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:15.630882 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:15.630837 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" podStartSLOduration=1.630823188 podStartE2EDuration="1.630823188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:25:15.629175094 +0000 UTC m=+3712.761834373" watchObservedRunningTime="2026-07-28 11:25:15.630823188 +0000 UTC m=+3712.763482467" Jul 28 11:25:15.786255 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:15.786231 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7456544f99-mzlcx_04ddf026-8ed6-4190-afaf-cedc08a77bcf/console/0.log" Jul 28 11:25:17.601602 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:17.601575 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nv8zj_0ff2fb62-a7e2-4295-aceb-6fdeef1993b9/dns/0.log" Jul 28 11:25:17.624216 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:17.624180 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nv8zj_0ff2fb62-a7e2-4295-aceb-6fdeef1993b9/kube-rbac-proxy/0.log" Jul 28 11:25:17.717827 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:17.717804 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9dld_591f7c33-2794-4cba-bc61-4b4ec073028a/dns-node-resolver/0.log" Jul 28 11:25:18.352994 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:18.352961 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5879f5564b-9vffh_08fb446f-5afe-4551-9e6a-5a21956bc677/registry/0.log" Jul 28 11:25:18.374751 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:18.374721 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qh9r_1e71134f-cb26-40d1-90e4-50a4a79e40e1/node-ca/0.log" Jul 28 11:25:19.513184 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:19.513154 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w2lzb_11e63ffc-3903-4e58-ba63-0ceac4f58f4c/discovery/0.log" Jul 28 11:25:19.676319 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:19.676296 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v57vq_2171fb23-48ec-41f9-a64d-a56007c59f83/payload-pre-processing/0.log" Jul 28 11:25:19.705323 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:19.705298 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log" Jul 28 11:25:20.618969 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:20.618928 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xp886_2d5d4cd7-0d96-471b-894c-fe57fd572b01/serve-healthcheck-canary/0.log" Jul 28 11:25:21.124104 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:21.124070 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2lsd5_453b6262-344b-42ae-b448-629bce3a1a62/kube-rbac-proxy/0.log" Jul 28 11:25:21.144984 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:21.144962 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2lsd5_453b6262-344b-42ae-b448-629bce3a1a62/exporter/0.log" Jul 28 11:25:21.165757 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:21.165706 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2lsd5_453b6262-344b-42ae-b448-629bce3a1a62/extractor/0.log" Jul 28 11:25:21.627498 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:21.627470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-74cfm" Jul 28 11:25:22.871563 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:22.871535 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6p989_1e5663e5-0741-4934-8dd8-5fae3ad0da7a/konnectivity-agent/0.log" limit=1024 Jul 28 11:25:24.199628 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:24.199598 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d7d4f6df-67vlx_9f933be8-6456-4d3f-928c-af17892ee591/manager/0.log" Jul 28 11:25:24.497451 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:24.497422 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-v2b8k_47faa125-f9cf-4372-8e33-e2b5d6fc2b39/manager/0.log" Jul 28 11:25:24.514140 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:24.514114 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-v2b8k_47faa125-f9cf-4372-8e33-e2b5d6fc2b39/copy-manifests/0.log" Jul 28 11:25:25.819445 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:25.819416 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w2lzb_11e63ffc-3903-4e58-ba63-0ceac4f58f4c/discovery/0.log" Jul 28 11:25:25.840906 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:25.840883 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d8dfbcf79-tvlhc_bda75dab-ae79-4485-a8b0-ee4385b6630d/manager/0.log" Jul 28 11:25:25.866302 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:25.866246 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ddnrq_55072a23-0aea-480c-a450-411bc7272c3a/openshift-lws-operator/0.log" Jul 28 11:25:25.944525 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:25.944503 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v57vq_2171fb23-48ec-41f9-a64d-a56007c59f83/payload-pre-processing/0.log" Jul 28 11:25:25.966536 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:25.966513 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log" Jul 28 11:25:27.468033 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:27.467999 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w2lzb_11e63ffc-3903-4e58-ba63-0ceac4f58f4c/discovery/0.log" Jul 28 11:25:27.625267 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:27.625236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v57vq_2171fb23-48ec-41f9-a64d-a56007c59f83/payload-pre-processing/0.log" Jul 28 11:25:27.644752 ip-10-0-130-55 kubenswrapper[2567]: I0728 11:25:27.644716 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ll5xj_e5e93679-4dc9-470d-89c9-a4ba29777ac7/payload-processing/0.log"